Vision Pro could gain an Apple Pencil-like controller

Either a very chunky Apple Pencil or someone can't draw to scale very well

Despite Apple emphasizing that the Apple Vision Pro does not require any kind of handheld controller, it is researching how to make an Apple Pencil for it that will have virtual brush tips, seen only by the wearer.

The Apple Pencil lineup has been accused of being confusing for consumers to pick from, but at least the whole range is similar. Apple's latest proposal looks like an oversized, chunky addition - but it's meant to be less a writing implement and more a handheld controller.

"With one illustrative configuration... [the device] is a handheld controller having an elongated marker-shaped housing configured to be grasped within a user's fingers," says the newly-revealed patent application, called "Computer System with Handheld Controllers."

Detail from the patent application showing an Apple Pencil with many controls

"A handheld controller with a marker-shaped housing may have an elongated housing that spans across the width of a user's hand and that can be held like a pen, pencil, marker, wand, or tool," it continues.

Rather than solely being used for, say, an iPad, Apple suggests that this device could be a controller for everything — a head-mounted device, cellular telephone, tablet computer, laptop computer, wristwatch device, a device with a speaker, or other electronic device..."

Where it is used with head-mounted device, such as the Apple Vision Pro, the patent application proposes that the marker could have a virtual tip.

"The head-mounted device or other device may have a display configured to display virtual content that is overlaid onto real-world content," it says. "[A] handheld controller [could have] a tip portion onto which a computer-generated paint brush head is overlaid."

Detail from the patent application showing the proposed device with two different virtual tips

Apple also suggests a more vague "computer-generated tool head," might be of use.

Internally, this controller could "include an inertial measurement unit with an accelerometer for gathering information on controller motions such as swiping motions, waving motions, writing movements, drawing movements, shaking motions, rotations, etc." It could also feature "wireless communications... tracking features such as active or passive visual markers that can be tracked with an optical sensor in an external electronic device," and more.

In typical patent application style, Apple's proposal is as broad-ranging as it conceivably could be, while remaining specific about concerning a "handheld controller [which] may have a housing with an elongated shaft extending between first and second tip portions." But the majority of the example illustrations show a chunky Apple Pencil, complete with a large and physical tip.

The patent application is credited to five inventors, including Yuhao Pan, who previously worked on a proposal to use ultrasonic sensors to authenticate user voices.

Follow AppleInsider on Google News

3 Comments

BirderGuy 27 comments · 2 Years

About 3 months ago

Based on virtually every analysis of Apple’s patent drawings, this is guaranteed to never appear in this form factor or even in one of its products. Usually the core technology in the patent ends up hidden in a future product in ways no one ever suspected. And so sorry that 3 versions of Apple Pencil is confusing to you.

Xed 2578 comments · 4 Years

About 3 months ago

BirderGuy said:

Based on virtually every analysis of Apple’s patent drawings, this is guaranteed to never appear in this form factor or even in one of its products. Usually the core technology in the patent ends up hidden in a future product in ways no one ever suspected. And so sorry that 3 versions of Apple Pencil is confusing to you.

1) The patent drawing isn't an exact rendering of the implementation in the device they are creating. Go back 100 years and you'll see that, but the system if very different now.

2) Based on this image can you tell me the 3 versions of the Apple Pencil? I can, which is better than most, but I can't tell you which devices they will work on. I'll have to google that... every... single... time. My feeling is that Apple cut corners in future proofing the Pencil since it was always going to be a small peanuts accessory which is why their cross usage is abysmal and confusing. Furthermore, as someone who is not a regular iPad user, but who uses their MBP all day, I have a real need for a good stylus on my trackpad. I would love for Apple to include a digitizer in the MBP trackpad.

tht 5377 comments · 23 Years

About 3 months ago

Xed said:

BirderGuy said:

Based on virtually every analysis of Apple’s patent drawings, this is guaranteed to never appear in this form factor or even in one of its products. Usually the core technology in the patent ends up hidden in a future product in ways no one ever suspected. And so sorry that 3 versions of Apple Pencil is confusing to you.

1) The patent drawing isn't an exact rendering of the implementation in the device they are creating. Go back 100 years and you'll see that, but the system if very different now.

2) Based on this image can you tell me the 3 versions of the Apple Pencil? I can, which is better than most, but I can't tell you which devices they will work on. I'll have to google that... every... single... time. My feeling is that Apple cut corners in future proofing the Pencil since it was always going to be a small peanuts accessory which is why their cross usage is abysmal and confusing. Furthermore, as someone who is not a regular iPad user, but who uses their MBP all day, I have a real need for a good stylus on my trackpad. I would love for Apple to include a digitizer in the MBP trackpad.

I can tell you which version of the picture Pencils will work with which iPads, off the top of my head. ;)

Apple designs its accessories as point solutions for specific products. It's an accessory, not a mainline product. So, the various versions of Apple Pencils are very much a product of the iPad they were meant to work with at the time. Why they do it this way? Who knows. I sometimes think accessories are the training ground for their junior designers and engineers. So, you get what you get, and the designers and engineers get valuable experience. There isn't a grand plan.

If you read the proposed patent, it's a handheld controller for a head-mounted device, and not an Apple Pencil. So, AI's title hits it right on the nose. Apple is actually claiming a size range for this device, at least for one of the independent claims: "the housing has a length between 130 mm to 160 mm and a diameter between 10 mm to 15 mm". The Apple Pencil is 8.9 mm in diameter.

Anyways, in essence, the patent is claiming a controller that is shaped like a stick, one end has tip with a force sensor, motion sensor, a vibrator, and various control buttons and stuff. So, essentially an Apple Pencil with a Taptic Engine as one example. It could have a scroll wheel in one end. It could consist of two halves that you take apart to two hand it. Other derivatives of a stick as a handheld controller.

Apple has not revealed how you scroll long lists in VisionOS, other than a pinch-sweep and hold perhaps. A handheld controller with a scroll wheel could do that. They will need to develop a VisionOS hand gesture language that stays consistent for a lot of this stuff first and foremost imo.