Windows

This was one of those adventures where you find a problem and it snowballs from there.

As part of my work on the Barrier project, we needed a way to build it with the Bonjour SDK from an online CI/CD service. Bonjour is used by Barrier to discover other barrier hosts on the same network. On Windows, the SDK files are needed. Other platforms natively come with the library or similar APIs already preinstalled.

But there’s an issue. The SDK is behind a login wall on Apple’s site. Rehosting it might be possible but it might violate some agreement within the installer. Luckily, the sources are open and are under very permissive licenses.

To save some work, I reused the XBMC project’s fork and made my fork of the SDK. I didn’t change any of XBMC’s changes, if any. I added a CI system to build the necessary libraries in the same manner that Apple did.

So, if you need a free, Cloud CI system accessible Apple Windows Bonjour SDK, look at the releases in my repo! The library files are there and users can fork my build to build the stubs their own way as well.

Links

• Azure Pipelines Build View: https://dev.azure.com/nelsonjchen/NC%20QEMU/_build
• GitHub Repo: https://github.com/nelsonjchen/nc-qemu/
• Releases Download Link: https://github.com/nelsonjchen/nc-qemu/releases
  • Is this outdated and not matching the latest release of QEMU? Contact me.

Changes

• Azure Pipeline to build x86_x64 target QEMU in MSYS2 on Windows
• Upgraded to Capstone with fixes that allow building in MSYS2.

About

“NC QEMU” is a very lightly augmented and experimental fork of QEMU to build on Azure Pipelines and with Windows Hypervisor Platform (WHPX) support for users who care the most about running the x86_64 target fast. This build only cares about the x86_64 target and other targets are not built.

The output is a zip file with some DLLs and qemu-system-x86_64.exe. This QEMU distribution can be run from a folder by itself. Alternatively, qemu-system-x86_64.exe can be dropped into QEMU for Window’s installation directory at C:\Program Files\qemu, replacing the existing version. Some features like USB network redirection might be missing though.

The pipeline provides a differently compiled QEMU compared to QEMU for Windows’s own build instructions which are cross-compiled on Linux. Notably, NC QEMU is built on a Microsoft-provided copy of Windows in MSYS2 with access to the Windows SDK headers for WHPX distributed by Microsoft. The MingW64 Toolkit which QEMU for Windows is built with has unfortunately currently not reproduced the WHPX headers in a free software manner. Users who want a QEMU for Windows with WHPX support but aren’t licensed nor want to accept the terms of installing the Windows 10 SDK are out of luck.

Another benefit is that this project provides a declaratively made and executable pipeline that builds QEMU for Windows. Fork, setup an Azure Pipeline, and adjust if needed. This can be considered a bit “executable documentation” for building a QEMU x86_64 target of this nature. The build logs are public and can be used as reference. For users having trouble building or configuring their systems to build QEMU, this reproducible setup can be quite useful.

Why QEMU+WHPX for x86_64

Absolutely not exhaustive:

• Acceleration is great. You can reach near-native speeds with acceleration.
• WHPX is native to the OS. No foreign kernel drivers/modules/extensions needed.
• It’s great for users who don’t have access to Intel’s HAX because they either want Hyper-V and/or AMD support. With AMD CPUs being multi-core bargain monsters, developers and power users on AMD are becoming more numerous.
• QEMU+WHPX boots Windows ISOs. HAX currently does not.
• QEMU is easier to hack on, script, and developed by many organizations, not one. WHPX support was contributed by Microsoft but is probably most useful for Google’s QEMU-based Android emulators!
• Users hacking on or working with QEMU in Windows can bring their work to Linux KVM accelerated systems easier. And vice-versa.

Packer WHPX workaround

The executables provided here do work for Packer. Simply add the directory with qemu-system-x86_64.exe to the PATH. Packer does not currently recognize whpx as a valid argument for the acceleration key. The workaround is to add it to qemu-args as a manual argument.

Additionally, the cpu argument of QEMU does not support cpu=host for whpx. Specify something supported manually.

References

• Orbital (PS4 Emulator) Build instructions: https://github.com/AlexAltea/orbital/wiki/Building

So I’m investigating running Windows XP inside libvirt in Ubuntu on my MacBook for continuous integration testing purposes with Jenkins. I tried for many hours to get it working but Windows would not boot up or it would get stuck on the NTLDR cannot be found issue. As it turns out, I did not enable “Enable hypervisor applications in this virtual machine”. I was under the assumption that I would not need to check that box as I wrongly assumed qemu would handle all the needed machine translation and that an error with bootup is not the fault of lacking the ability to use a CPU feature to emulate something or the disk emulation being borked. With five different ISOs of various pedigrees, I tried installing Windows XP with virt-manager and all of them failed to bootup with various disk image formats like qcow2 or raw. It was only until I checked that checkbox did it then work.

That’s my surprise for today. I really wished I checked that checkbox earlier! It determines if your machine can bootup and only using qemu without kvm will not cut it for booting Windows XP inside libvirt inside Ubuntu inside VMware Fusion.

I paid Swish $25 and $8 shipping for this two months ago. I think I had my share donated to the Khan Academy.

It’s the “Windows Quickstart Kit for Mac Developers”. It includes:

• A full Windows 8 Pro license
• A Parallels Desktop license
• A USB Stick with an ISO on it with a real Microsoft Certificate of Authenticity. Genuine Windows!

I’m not sure why Microsoft saw fit to spend two months packaging a “quickstart”. Why Microsoft did not see fit to provide this quickstart through digital distribution is beyond me. They could have emailed a link to some keys and ISOs and maybe even let a distributor like Digital River do this. Instead, they saw fit to contract this work out to a physical media distribution company. It seems with these charity-ware deals that the best way to donate money is to allocate a large chunk to some physical distributor.

I’m so stoked to try and develop for Windows Phone. I’m going to virtualize Windows 8 and develop a native app for a platform on which I have no way to run natively. I’m also going to jump over some chairs in celebration.

Nuh uh, virtualization is cheating and is really anti-dogfood. If I have to boot up a RAM hungry OS to RUN the app, it’s a failure. If I have to develop the app inside a virtualized OS, that would be even more of a failure since I would have to bear with the performance and non-nativeness penalty. There’s no way a good app can come out with so many barriers like that. Google realized that its Eclipse toolkit was a problem and switched to IntelliJ IDEA to reduce their barriers and gain an awesome IDE in the meantime. I highly doubt Microsoft will provide a free cross-platform and light SDK. The technical barriers are just too high. It’s totally not like their Xbox where game developers are used to putting up with shit like that. Mobile developers are in general more finicky and prefer native tools. I’m not sure if this is still the case now but the Android team saw fit at one point to maintain the Android compilation toolchain for the entire OS on OS X. You can compile a Linux system on OS X!

With that said, this cool cardboard box is all that remains. It would be well designed if it never existed. It’s way too late for that though.

It’s made up of two big pieces. After ripping off the plastic wrapper, there’s a windows logo cutout sleeve thing to keep the box with a flap closed. It’s a pretty retail package for something that is only sold online.

When you open the box, you have a welcome card with some very basic instructions on the back on the left and some coupon card things on the right.

Beneath the welcome card is a USB drive. It has a Microsoft certificate of authenticity on it, so you know its real and has real Microsoft binary bits. There’s also a tracking ID on it too. The Microsoft logo is embedded in plastic on the back and not some cheap printer job. It’s certainly done with style to make the USB drive seem to be actually worth something.

On the USB Drive is just an ISO. There are no other files The drive itself isn’t bootable. It is formatted with NTFS though. At the very least, it’s readable on all platforms. It also enumerates as a device of the name Windows 8 Pro in system profiler. Quite a Matroska doll. I guess having an ISO is easier than mounting a USB drive in a virtual machine in terms of instructional material.

The cards on the right had keys on them. They’re business card sized and feel like them too.

On the back were some keys. Why they felt the need to kill all these trees and grow soy beans to produce the ink for printing on said dead tree material to wrap around these numbers is beyond me.

It took two months for these two alphanumeric strings to get here. The lack of timeliness on delivering these two codes and an ISO does not look well for Microsoft’s Windows efforts at all. It would take 30 minutes to illegally obtain these materials. 30 minutes is far less than two months. Oh, I almost forgot, they don’t even include Parallels in the package. You will have to go download that from Parallel’s site. What a “quickstart” indeed.

A few days ago, I decided to pull the trigger and install Windows on my Retina MacBook Pro at a LAN Party. It’s a perfectly fine machine with a decent graphics card and CPU. I started Boot Camp Assistant and followed through with the instructions to install from the USB drive using the ISO of Windows 7 I had on my Desktop. If you follow Apple’s steps to the letter, your installation of Windows will never proceed.

If you do follow the steps, you will most certainly get an error of “Setup was unable to create a new system partition or locate an existing system partition. " when you select the BootCamp partition that you had just formatted in the Windows installer on your Mac. This can occur on PCs too but basically the issue is that the USB drive is initialized as another valid bootable disk. If you were to Google this, you will find solutions online that involve yanking out the USB drive and reinserting it before pressing “Install Now” on the Windows 7 installation screen. This “Install Now” screen is a big centered button that prompts your to install now and the arrow is set inside a blue circle that’s like a gem.

This is where Apple’s helpfulness can get in your way. By default, Boot Camp Assistant copies the Boot Camp support files to your USB drive. It also adds an Autounattend.xml file to the root of your drive to automatically install Boot Camp support drivers and utilities in Windows. With Autounattend.xml present, you will not have the opportunity to yank out and reinsert the USB drive before continuing. Instead, the setup just proceeds as if that button was pressed. While this is a good file for disc installations, it causes a massive problem for USB installation which is most likely to be the case for all new Macs from now on since they do not include disc drives by default. If you try to ask Apple to ‘fix’ this, they will just blame Microsoft which admittedly isn’t blame free either.

To solve this problem, move or delete Autounattend.xml on the root of the USB drive and proceed with the normal steps afterwards . After Windows is done installing and since Autounattend.xml was nullified, you will have to run the Boot Camp support installer from the BootCamp folder on the USB drive.

It’s a shame Apple’s instructions just do not work. The fix required would probably mean removing Autounattend.xml functionality for USB Drive installations. In the meantime, this worked for me.