Apple's patented physics-based iPad GUI translates file size into mass, supports intuitive gestures

The U.S. Patent and Trademark Office on Monday granted Apple U.S. Patent No. 8,839,150 for "Graphical objects that respond to touch or motion input," which describes a hybrid mobile operating system and corresponding graphical user interface that responds to touches based on implied physical forces like mass, gravity, friction or drag.

As described by Apple, the technology can be applied to a tablet's UI, though the system described is more full-featured than iOS in that it handles files and folders. For example, instead of arranging apps in rigid grids, icons can be placed and selected in free space, a method familiar to OS X users. Unlike Apple's current desktop operating system, however, the patent relies on touch gestures as a main form of input.

At the heart of Apple's patent is organic user interaction facilitated through graphical animations driven by sensors, touch gestures and intelligent design. The document offers a first example that involves selecting a batch of folders.

In a usual desktop operating environment, selecting one or more UI elements implies using a mouse to create a highlighted box around one or more icons representative of folders, files or apps. With Apple's invention, a user draws a circle around the same two-dimensional assets, but when they lift their finger, the elements are transformed into graphical objects.

While the representation can be any number of animations or other graphics, the document describes a floating bubble or sphere that appears detached from the background UI. In this way, a user can quickly distinguish what folders are active, or selected.

In addition, by leveraging onboard sensors like accelerometers and gyroscopes, the system can add physical properties like mass to the floating ball, which correspond to device motion. In one embodiment, the ball can bounce off screen borders or move in accordance to simulated drag and friction.

Further, the mass of a graphical asset can correspond to its data size. For example, a large file or folder will create a graphical object of with proportional mass. The larger the file, the larger the mass, which in turn dictates how the graphical asset responds to user interaction. In some embodiments, a large file will move more slowly as its mass is larger than other files in the UI.

Another interesting concept tied to physical file attributes is how these graphically represented files respond to actions like transferring, compressing and sorting, among others.

In the illustration at the top of this article, a user makes a "pouring" gesture with one device onto another device connected via Bluetooth, Wi-Fi or similar wireless technology. Again, based on file size, the assets obey the rules of gravity and "fall" into an opening on one device, then show up on the recipient machine's display as if being poured.

As an example for compression and file archiving, users can make a pinching gesture around a selected asset, which shrinks the graphical representation once the operation is completed. Alternative embodiments include tags, text or file replacement confirming the compression action and displaying the new file size.

The document again points to a graphical representation of data as mass, this time applying it to file sorting. For example, users can shake their device to create a "snow globe" effect that results in large data files sinking to the bottom of a UI, while smaller files remain at the top. Similar sized files and folders are arranged in bands.

While the invention would be an interesting addition to an Apple product, the property appears to be aimed at a GUI that lies somewhere between a touch-capable version of OS X and a more feature-laden iOS. Since Apple seems reticent to join the two operating systems, the patented technology will likely remain a curiosity for the foreseeable future.

Apple's graphical objects patent was first filed for in February 2010 and credits Nicholas V. King, Brett Bilbrey and Todd Benjamin as its inventors.