Apple patents 3D gesture UI for iOS based on proximity sensor input

The U.S. Patent and Trademark Office on Tuesday published an Apple patent for a method of generating and manipulating a three-dimensional object on a computing device, with the process controlled by special gestures made above a touchscreen's surface.

Apple's U.S. Patent No. 8,514,221, titled "Working with 3D objects," describes a graphical interface that enables a user to "generate and manipulate 3D objects using 3D gesture inputs." More specifically, the interface can be a computer assisted design (CAD) application running on a computer with a touch-sensitive surface, such as an iPad.

The document refers to a device that can detect the location of fingers with a combination of capacitive touch sensors and proximity sensors embedded in the display. These two components can be separate, or the capacitive sensors themselves can act as proximity sensors by measuring the capacitance of a nearby finger.

The patent deals with the creation of 3D objects "extruded" from 2D images. In order to generate the 3D renderings, a user can manipulate a 2D object with conventional touchscreen gestures, such as pinching. Instead of continuing with established iOS gestures, the invention deviates by introducing a third axis of control.

For example, to extrude a triangular prism out of a triangle, a user can touch the two-dimensional object in three places and "pull" or "lift" up, away from the screen's surface. Here, a user can hover for a predetermined amount of time to trigger the extrusion, which will generate a triangular prism with a cross section corresponding to the 2D triangle and a height proportional to how far a user "pulled" their fingers from the screen.

To indicate the end of a 3D gesture input, or deselection of an object, the application can be programmed to detect any number of 3D gestures, such as a user spreading their fingers or quickly moving them away from the screen. An object can be reselected by touching and holding on the screen for a certain amount of time.

Adding to the 3D interface, users can augment the "pull" mechanism with other motions to create various shapes. For example, instead of the triangular prism described above, a pyramid can be generated from a 2D triangle touching the screen with three fingers and pulling away from the surface while pinching. Alternatively, a user can use a pull and spread action to create a frustum, or the top section of a cone or pyramid that is left behind after being cut off from its base.

Further, modifications can be performed with "pinch and pull" or "pinch and push" gestures, which extrude or push in a portion of a 3D object, respectively. In another embodiment, objects can be rotated around the z-axis with a twisting motion.

One interesting application described in the document is dubbed "sculpting" mode. This embodiment treats the 3D object as if it were made of clay, or some other easily malleable material. Depending on finger movement, a user can make indentations, stretch, or squeeze the shape so that one area is made smaller and another larger. A "pinch-twist-and-pull" gesture can also be used to break off a piece of the clay rendering.

In yet another use scenario, textures, colors and other surface attributes can be selected and tuned by hovering over a screen's surface. These gestures are largely made perpendicular to the device's display, with selected attributes corresponding to height away from the screen. For example, an object can be made brighter when a finger hovering over a "Brightness" UI region is moved farther away from the display. Slide bars, buttons, and other assets may act as alternative controls.

Finally, the patent makes mention of using 3D or stereoscopic glasses to make the experience even more engaging. Also noted are further UI implementations such as iconography, graphical assets and general usability considerations.

Technology similar to that described in Apple's patent is just now making its way to the mass market in the Leap Motion Controller. As Leap is a computer peripheral, use cases are somewhat limited. For now, the system relies largely on third-party software makers to create interesting apps that make use of the motion-based control scheme.

If deployed at the system level of iOS, Apple's tech could turn an iPad into a hybrid touchscreen/motion controlled device, bringing a more direct control experience than Leap can offer. In some ways, a possible iPad implementation is to Leap as what the touchscreen is to the mouse.

Apple's 3D object creation and manipulation patent was first filed for in 2012 and credits Nicholas V. King and Todd Benjamin as its inventors.