Apple wants to extend AR into the real world with image projectors

Apple could be planning for a future where we no longer need screens or an Apple Vision Pro to do our work, or to interact with AR.

Right back at the start of the Mac, an issue was training users to get that when they moved the mouse, the cursor would move on the screen. While we're all used to it now, it is a conceptual leap that was new then.

And now all these years later, Apple is still concerned with any dissociation between physical reality and computing.

"Interacting with displayed images that have no connection to real-world objects may leave the user feeling removed from the user's real-world environment," says Apple in the newly-granted patent, "Electronic Devices With Projectors."

The patent reads as if it is just one part of a set, and this is the part that is practically obsessed with how projectors could create real-looking shadows around objects. "[One] or more image projectors such as projector that projects images onto a surface," is about the only reference to what the projectors would display other than shadows.

"An electronic device may include a projector for creating the appearance of animated shadows on a surface," it says. "The animated shadows may be created by projecting ambient-light-matching illumination onto the surface that blends in with the surrounding ambient light."

"Select pixels in the projector may be turned off' so that one or more unilluminated regions are created within the surrounding illuminated region," it continues.

So the majority of the patent appears to be about calculating realistic shadowing by turning off pixels next to objects that the Mac or other device is projecting. It's possible that it would also create shadows when a sensor detects "hand movements made near the surface, hand movements made with objects on the surface, hand movements made near the shadows, etc.)."

Reading the 18 pages of the patent, with its 20 claims and 8 illustrations, it's hard not to be reminded of the monster chess game in "Star Wars." The patent does not, though, concern itself with what would be projected, or what the use of such projection would be.

It does very much concern itself with what would actually do the projecting. Apple's patent directly lists over 30 different possible devices ranging from the obvious, such as Apple Vision Pro, to the less obvious, such as a desk.

This is clearly a case of Apple doing what all patent writers do and make a proposal specific enough to be patented, but broad enough to cover as wide a possible range of implementations as is conceivable.

A projection system could also contain sensors to spot hand movement, and create realistic-looking shadows across the AR surface

So it's most likely that the projector idea is really for "a laptop computer, a computer monitor..., a tablet computer, a cellular telephone, a media player... [or] a smaller device such as a wrist-watch device."

However, among the many other options there is mention of "a pendant device... a gaming device, a navigation device... voice-controlled internet-connected speaker... or equipment embedded in lamps, tables, chairs, desks, other furniture."

There's also reference to such projector displays "mounted in a kiosk or automobile," or even a remote control.

Why Apple may be interested in projectors

With such an enormous range of listed applications — long on names, if also short on any detail — it's hard to see just what products might arise from this, assuming any do.

However, if the application and the design is hard to follow, the intention is quite clear.

"Computers and other electronic devices may sometimes include displays," says Apple at the very start of the patent. "A display may present images to a user."

"With touch-sensitive displays, a user may interact with the images by providing touch input on the display," it continues. "Limitations may arise with traditional displays."

And then we get it: "[for example] the user may wish to interact with real-world objects in the user's environment in addition to or instead of interacting with images on a display."

This is all about making it possible to work and have natural interactions with computer-generated elements without requiring the use of screens.

The patent is credited to two inventors, Christopher J Verplaetse and Clark D Della Silva. It appears to be their first patent for Apple, but Verplaetse has previously been credited on a patent concerning "Motion-based input system for handheld devices."