Future Apple Vision Pro may get Bob Ross-style virtual painting tools

We recently reported on rumors that it is testing the Apple Pencil for AR and VR, but it looks like the company hopes to eventually ditch the stylus entirely.

Apple's proposed technology is based on two patents allowing users to draw or paint in augmented reality. Users would be able to paint on a virtual canvas. However, the technology also allows the creation of art in the real world, where artwork could be superimposed over actual elements.

The first patent application, "Stylus-based input system for a head-mounted device," describes exactly what the rumors suggested: Apple is working on using a stylus to interact with a virtual world.

While the Apple Pencil seems the natural choice for an input device, the patent describes a new device with a light emitter (likely infrared) to allow the headset to track the device's position and orientation.

A second patent "Techniques for enabling drawing in a computer-generated reality environment" takes this even further.

This illustration shows Apple's plan to track a user's hand. Pinch your fingers as if holding a paintbrush, and one will appear superimposed over your hand. Switch tools and the Apple Vision Pro would change the display of what you're holding.

While it wasn't mentioned in the application, adding gestures for different media would be easy. You might be able to virtually spray paint using the gestures of holding a can and pushing a button or work in finger paint by extending your fingers.

Apple wants to create a virtual simulation of an artist's workflow with unique gestures for selecting colors. With this technology, you could pick and mix colors by flipping over one hand and "dipping" a virtual brush into virtual paint. The experience would mimic artists holding a paint pallet in one hand and the brush in another.

Drawing in a 3D space isn't unique. Anyone with a headset like the Quest can use Tilt Brush or other art programs to create virtual artwork.

However, those programs use controllers and are, at best, awkward. Pushing a button isn't a natural way to draw or paint, and the controllers get in the way of the experience.

In addition to virtual painting, Apple's patent suggests that audio feedback could let a user hear the brush strokes or pencil lines on paper. Immersive experiences like this could transform artwork creation in AR/VR.

Audio feedback could also be used to take virtual art classes. Put on a headset, listen to soothing music, and follow the prompts of a painting instructor. The result could be an art class in a headset.

We can imagine this being combined with other Apple patents for AR/VR, which would allow multiple users to view and interact with the same virtual objects, even if they are not in the same space. For example, you could take a class with a real instructor and paint alongside them on the same virtual canvas.

As with all patent applications, there's no guarantee that these technologies will come soon, or even ever. But combined with the rumors that Apple has been testing the Apple Pencil in Apple Vision Pro, it's safe to say that some of this technology will be available very soon, perhaps as soon as WWDC 2024.

Apple's patent for using a stylus for input is credited to Killian Moore, the lead applicant on numerous AR/VR patents filed by Apple. The patent for drawing in AR/VR was created by Edwin Iskander, Senior Manager of Apple's visionOS.