Wednesday, August 11, 2021

Customizing MacOS guest VMs in Parallels 17 on Apple Silicon

Those of us that need to test and package software for MacOS on Apple Silicon (aka M1) have spent the past many months bemoaning the lack of virtualization options for MacOS on Apple's new flagship hardware platform. Modern build environments lean heavily on VM images (and/or containers, where available) to ensure safety, isolation, and repeatability. Unfortunately, since the boot process for Apple Silicon MacOS uses a bootloader borrowed from iOS, the typical EFI-based bootloader that's used to boot Intel Mac images won't work. Since its release, Apple Silicon has not offered any virtualization support for MacOS itself, which also explains the lack of things like GHA/AZP build workers for M1/Apple Silicon. So when Apple quietly announced that the upcoming MacOS Monterey offered MacOS guest support, there was much rejoicing.

After installing Monterey Beta4 on my M1 Mac Mini, I spent a long evening playing around with the new Parallels 17 support for MacOS guests. It's definitely *extremely* early days- most of the support for common Parallels features isn't wired up for MacOS guests, as they use a completely different set of disk images and tools that are a very thin wrapper around Apple's new Virtualization framework. Most of Parallels' great automation and command-line tooling is currently completely unaware of the new MacOS guests on Apple Silicon. If you're going through the UI "front door", it doesn't appear possible to customize the VM in any way (even its name in Control Center; as of this writing, creating multiple Mac guests names them all "macOS"). The bigger issue that I set out to solve is the inability to customize the default disk image size of 30GB to make the VM useful for simple development tasks- the default size is too small to even install the XCode command-line tools. While none of the usual Parallels tools or APIs appear capable of customizing an M1 MacOS guest or its images, a bit of poking revealed a couple of command-line tools buried in the Parallels 17 package that will allow some basic customization of new VMs using undocumented args.

The Parallels tool that wraps the Virtualization framework APIs for creating a new VM image from an Apple IPSW archive can be found at:

/Applications/Parallels\ Desktop.app/Contents/MacOS/prl_macvm_create

It has a couple of modes; calling it with `--getipswurl` will try to find a working download link for a compatible IPSW package to use to seed the new image, though I prefer to just use the list maintained by MrMacintosh. Regardless where it comes from, downloading an IPSW image is the first step to creating a new VM. When you're using the Parallels "New" button, most of the time is spent on the IPSW download, so if you want to make a lot of VMs, downloading and reusing the IPSW for each VM will save a lot of time and bandwidth (as they're ~13GB each). 

Once you have an IPSW image locally, run prl_macvm_create with the path to the IPSW, and the path where you want the VM image to live (the default is under '~/Parallels/macOS 12.macvm'). This is also your chance to increase the default disk size by adding `--disksize` and the desired disk image size (in bytes). If you omit this arg, your VM image will be created with a tiny 30GB disk that can't do much more than run the OS itself and allow for some small software installations. If you're planning to install XCode, I'd recommend at least 60GB.

Here's an invocation that uses a local copy of an Monterey IPSW image to create a new VM at ~/Parallels/devmac1.macvm with a 60GB disk:

/Applications/Parallels\ Desktop.app/Contents/MacOS/prl_macvm_create ~/Downloads/UniversalMac_12.0_21A5294g_Restore.ipsw ~/Parallels/devmac1.macvm --disksize 60000000000

Assuming all's well, it should create a few image and config files under the path you specified, followed by `Starting installation.` and some progress messages. This process usually completes in a couple of minutes.

Once you're greeted with `Installation succeeded.`, your VM should be ready for its first boot. You can use "Open" in the Parallels Control Center to do this (any directory ending with the `.macvm` extension should be visible there) if you want it to behave as if you'd created it in Parallels, or if you want to run the VM directly from the command-line (which has some advantages), you can use

/Applications/Parallels\ Desktop.app/Contents/MacOS/Parallels\ Mac\ VM.app/Contents/MacOS/prl_macvm_app

With no args, this will run the default VM at '~/Parallels/macOS 12.macvm', or you can pass the `--openvm` argument to run any VM you wish.

Here's an invocation that runs the VM I created above:

/Applications/Parallels\ Desktop.app/Contents/MacOS/Parallels\ Mac\ VM.app/Contents/MacOS/prl_macvm_app --openvm ~/Parallels/devmac1.macvm 

This runs the VM inside the launched process, so Ctrl-C'ing the command or otherwise killing that process stops the VM. This is definitely a feature in my book for ephemeral VMs; it makes it pretty trivial to manage the running worker VMs in a CI environment by just starting the VM process and hanging onto its handle, signaling/killing it when you're done.

Side note: there are several OSS projects (eg, KhaosT's MacVM) that also wrap the calls to the Virtualization framework to create and run new MacOS guests under Monterey. The big win for Parallels right now is with its single command to create a new image. The open source image builders that I've seen will call into the virt framework to create a blank VM running in DFU recovery mode, but then require you to use Apple Configurator 2 to load the IPSW yourself into the VM. It works fine, but definitely less convenient for automation than what Parallels has rolled up into a convenient one-stop package, and I assume much of the rest of the Parallels value add from their excellent automation will come with time.

One thing we need right away is cheap throwaway VM clones; fully realized 60GB disk images are very expensive to copy around, run for a couple minutes, then delete and repeat. Thankfully, APFS'  copy-on-write cloned files (created by cp -c on an APFS-formatted filesystem) fit the bill perfectly with the disk image files that Apple's Virtualization.framework uses. Once you've configured a VM image to include whatever tools and startup behavior you want, simply shut down the VM, and copy the entire VM directory as many times as you'd like for (basically) free, eg:

cp -c -r ~/Parallels/devmac1.macvm ~/Parallels/cloned_ephemeral_mac.macvm 

The filesystem will only record changed blocks in the copied files once the VM boots up and starts doing writes. Once the clone directory is deleted, so are all the filesystem changes made under it.

Virtualization support is still quite early in the Apple Silicon ecosystem, but now we've got at least the very basic tools to do what's needed. Thanks Apple, Parallels, and all the OSS folks out there taking this stuff apart!

Another random side note: just for giggles, I tried creating a VM with a Big Sur 11.5.1 IPSW (nice to build against a released + supported OS), but it says "prl_macvm_create[10407:185738] No supported Mac configuration." - I assume there's some extra magic in the package required to allow it to be virtualized, so at least for now, it looks like Monterey+ is the only option for guest VMs.