Getting started with Apple Vision Pro developer software

Apple released its visionOS SDK and developer tools on Wednesday, along with more info about developing for Apple Vision Pro. Here's an overview of what it is, what it can do, and how to get started.

Apple's SDK for the Apple Vision Pro is included in the second Xcode beta. It includes Command Line Tools for Xcode 15 beta 2, Font Tools for Xcode 15 beta, and the visionOS 1 Beta Simulator Runtime, which is packaged as a .dmg disk image file.

To get the new SDK and Xcode beta files, go to developer.apple.com/download/ and log in with your Apple ID. These tools are included in the new free developer membership.

How to install Apple Vision Pro developer software

You'll need the Xcode 15 beta 2 download, which contains the Xcode 15 beta 2 app itself, macOS 14 beta 2, and assorted simulators. You'll also want the Command Line Tools for Xcode 15 beta 2, and the Xcode 15 beta 2 which also contains macOS 14 beta 2.

If you haven't previously, also download the Font Tools for Xcode 15 beta at the bottom of the page released on June 5th.

When downloading Xcode 15 beta 2, be sure to check the visionOS beta 1 checkbox at the bottom of the list, which weighs in at a whopping 6.85GB:

Note that in all cases you do need the iOS download components as well.

This provides the what Apple is still calling the xrOS 1.0.simruntime file on a .dmg which you import into Xcode to install the visionOS simulator (more on this below).

Additional downloads for Apple Vision Pro developer software

The learning curve for visionOS can be quite steep - crossing several computing disciplines in addition to mere programing. These include 3D object and scene design, Augmented Reality (AR), materials and surfaces, and lighting and rendering.

In addition to the Xcode pieces, you'll want to learn and grab a few additional items used by Apple's AR ecosystem:

1. ARKit 6
2. RealityKit
3. Reality Composer app
4. AR Quick Look plugin for 3D object file previews

Most of these are available on Apple's developer site at developer.apple.com/augmented-reality/

The SDK also includes Reality Composer Pro which was promised at WWDC 23 in the visionOS presentations. Reality Composer Pro is an app designed to help prepare and preview 3D content for visionOS.

As part of the SDK release Apple also released Reality Converter beta which allows you to convert between AR file formats such as .obj, .gltf and .usd and convert them all to standard USDZ format. You can also change some texture properties in Reality Converter beta.

There's also an additional .dmg download on the AR tools page called USDZ Tools which is a set of Python scripts for USD conversion. All of Apple's AR tools can be found on the main AR page at developer.apple.com/augmented-reality/tools/

The AR Quick Look plugin can be found at developer.apple.com/augmented-reality/quick-look/ along with samples.

With all this installed, you can now also use Unity's 3D development tools to develop Apple Vision Pro apps. Visit the Unity site at unity.com.

Apple Vision Pro SDK -- File and graphic formats

Designing and writing your visionOS app is just one part of visionOS development. The other part is mastering 3D models and spaces.

After all, Apple calls the Vision Pro an AR headset. As such, your visionOS apps are presented to the user in 3D space in the Apple Vision Pro device.

You'll also need to be familiar with Pixar's Universal Scene Description and USDZ files, which contain descriptions of 3D objects.There are a number of resources to learn about USDZ files. For instance, NVIDIA's developer site also has a good discussion of USD.

In most cases you'll need to convert your 3D content to USDZ for use in visionOS.

In additional to USD, you'll also want to learn Industrial Light + Magic's MaterialX standard - an open-source universal 3D object material standard first created in 2017 for the film Star Wars: The Force Awakens.

Materials refer to the surfaces of 3D objects and what those surfaces look like. In real world terms, this means materials like wood, metal, or glass that they are simulating.

Apple Vision Pro SDK -- User experience and interface

There are four modes of presentation supported in visionOS:

1. Windowed apps
2. Volumes
3. Spaces
4. Immersive apps

Windowed apps act essentially like iOS and iPadOS apps in visionOS's main Shared Space, except that you can fling them to the side - kind of like you can in macOS with Stage Manager.

Volumes are basically SwiftUI scenes embedded in windowed apps which display RealityKit or Unity 3D objects in parts of windows. Users can reposition windows and volumes in the Shared Space wherever they like.

Spaces are immersive "Full Spaces" dedicated to one app where only one app's content can appear. Inside a Full Space an app can display windows or 3D content, or it can display an entire 3D rendered scene such as a game or simulation, immersing the user in another world.Apple calls this "Unbounded 3D content."

As a developer, in visionOS you can allow the user to transition between any of these states and back whenever you like.

Installing the Apple Vision Pro SDK and simulator

We won't delve step-by-step into how to install the visionOS simulator in Xcode since it's already documented in the Xcode developer pages. First you'll want to install everything from the first Xcode 15 beta mentioned above, in particular the Xcode_15_beta_2.xip file which is a signed .zip file containing the Xcode 15 beta app.

After you install the first Xcode beta, download and install the command line tools, and the second beta.

You have to do this in that order since if you use the web install method (mentioned below) it requires the Xcode beta app to be installed first in order to work.

The entire process and all its options are described on the Xcode Devices and Simulators pages at Installing and managing Simulator runtimes.

The visionOS simulator download is substantial at around 7GB.

There are several ways you can install the simulator:

1. Install Simulator runtimes from the Apple Developer website.
2. Install Simulator runtimes during first launch.
3. Install a Simulator runtime from the Xcode run destination.
4. Install and manage Simulator runtimes in settings.
5. Install and manage Simulator runtimes from the command line.

The first option requires you leave all the downloaded components and the manifest files in the Downloads folder in your user folder since Xcode looks for them there. When they finish dowloading Xcode automatically installs them from that location.

It's probably best to use Safari for this option. You'll need to log in to the developer site using your Apple ID for this option.

The second option -- install on first launch -- only works the very first time you launch Xcode on a new installation. It asks you which simulators you want to install and if you're running the Xcode 15 beta on macOS 14 beta, those options will include the visionOS simulator.

To install the simulator manually from within Xcode go to Settings->Platforms. Click the Get button next to each simulator you want to install. Be sure you have enough disk space first for each one.

To install the simulator from the command line in Terminal you'll first want to run Xcode and when prompted the first time, install the additional Xcode tools. This installs a command-line tool called simctl. You can get more info on simctl in Terminal by typing:

simctl and pressing Return.

You can also install the command line tools by downloading the Additional Command-line Tools package mentioned above and running the Command Line Tools.pkg installer contained on the .dmg in the Finder.

The simctl commands required are listed on Apple's Xcode developer pages:

xcode-select -s /Applications/Xcode-beta.app

xcodebuild -runFirstLaunch

xcrun simctl runtime add "~/Downloads/visionOS_1_beta_Simulator_Runtime.dmg"

Note that the xcrun simctl runtime add command requires the simulator .dmg file itself in the Downloads folder to install the simulator, so don't open the .dmg directly.

You can also install one or more simulators by using the simpler xcodebuild command, and either the -downloadAllPlatforms or a specific simulator option.

For example, the iOS simulator can be downloaded and installed by entering:

xcodebuild -downloadPlatform iOS

Apple Vision Pro SDK -- Getting started

The visionOS SDK is similar to Apple's various other OS frameworks and developer tools. The basic parts you'll need to know are Xcode, SwiftUI, RealityKit, ARKit and TestFlight - and some 3D modeling tools.

There are two major 3D views you can use in visionOS for 3D objects and scenes: ARView and RealityView (which is still in beta).

There are a variety of other beta objects you'll want to check out including Model3d, RealityRenderer, SceneRealityCoordinateSpace and RealityCoordinateSpaceConverting.

Most visionOS apps and views are built using SwiftUI and Swift classes. Some of the newer objects don't have Objective-C versions.

In any existing iOS or iPadOS Xcode project you can add a new visionOS target to the existing project, or build an entirely new app by creating a new Xcode project based on visionOS from the Xcode's template chooser.

To do so, in Xcode go to File->New->Project and click the visionOS icon in the template chooser. Click Next, name the project, set its options, then click Next again and save it to disk by clicking Create.

You can add a new visionOS Target to your existing iOS or iPadOS Xcode project by selecting the project icon on the left side of the Xcode project window, then selecting the project name under "Project", then selecting File->New->Target from the File menu:

When the Xcode template chooser appears, select visionOS from the tab bar at the top just as you would if you had created a new project. Set the new target's details in the next sheet, then click the Finish button - a new visionOS target will be added to your existing Xcode project.

Apple Vision Pro SDK -- App compatibility

Apps using iOS, iPadOS, and visionOS can share many, but not all of the same technologies. To port your app to visionOS you'll need to focus on checking your apps and converting any incompatible parts to compatible or new visionOS technologies.

For example if you use Metal in your iOS or iPadOS apps, you can use Metal in visionOS, but only in immersive mode since Metal relies on sampling pixels and if you were to use it in windowed mode Metal might sample the user's background - potentially disclosing sensitive information when samples are taken from the environment.

In addition to brand new visionOS apps, most iOS and iPadOS apps can be adapted to visionOS. visionOS was designed this way to make it easy to port your existing apps.

When developing for Vision Pro using iOS or ARKit apps, you have several choices: you can see if your app is already compatible with visionOS, you can make your existing app work with visionOS, or you can make a brand new ARKit app which works with visionOS.

In the visionOS documentation, there's a section titled iOS migration and compatibility. In this section there are three subsections:

1. Checking whether your existing app is compatible with visionOS
2. Bringing your existing apps to visionOS
3. Making your existing app compatible with visionOS

Apple's main visionOS documentation page hosts most of the information you'll need to get started. It also has a section covering the Xcode visionOS simulator.

We'll dive deeper into the technical aspects of building visionOS apps in a future article.

Developer labs open in July 2023

Apple also announced they will be opening developer labs in several major cities around the world in which developers can test their apps on prototype Vision Pro hardware for compatibility and robustness. And, in July, developers can apply to purchase developer's kits containing the hardware.

Over the course of the next nine months -- and maybe more, depending on actual release date -- we'll dive deeper into porting and creating apps for visionOS, and what the platform means to everyday users. For now, though, Apple's expansive developer documentation contains huge amount of new material to learn.

After all, there's a reason Apple announced the hardware at the 2023 WWDC for a 2024 release.