Future iPad or iPhone could both recognize and display handwriting

Apple is continuing to investigate handwriting recognition for touch screens, but the latest research also covers displaying handwriting on an iPhone or iPad screen.

Apple has regularly filed patents regarding different aspects of handwriting recognition, but a new one is equally concerned with both inputting handwriting and displaying it, too. As well as actually recognizing and reacting to a user writing on the screen, the device may respond with simulated text instead of the usual typewritten one.

The aim is to make using computing devices more natural, and specifically to cut down on the number of mouse or trackpad clicks and taps for selecting and manipulating objects on the screen.

"[Traditional methods] for performing these manipulations are cumbersome and inefficient," says Apple in "Device, Method, and Graphical User Interface for Simulating and Interacting with Handwritten Text."

"For example," it continues, "using a sequence of mouse based inputs to select one or more user interface objects and perform one or more actions on the selected user interface objects is tedious and creates a significant cognitive burden on a user. In addition, these methods take longer than necessary, thereby wasting energy. This latter consideration is particularly important in battery-operated devices."

In outlining the patent's claims, Apple describes the sequence of events that a user will see.

"A device receives a user input that corresponds with a sequence of characters," it begins. "In response to the user input, the device displays simulated handwritten text that includes varying the appearance of characters in the simulated handwritten text based on variations in handwritten text of a respective user."

The detail of the patent is concerned with how a system can respond to different pen strokes, including line drawings, and variations on how characters are written.

These variations are not only recognized, but used as the basis for how the device's displayed handwriting varies in return.

"In response to receiving the user input, the method includes displaying, on the display simulated handwritten text that includes varying the appearance of characters in the simulated handwritten text based on variations that were detected in handwritten text of a respective user," continues the patent.

Detail from patent showing a stylus writing on an iPad

Beyond shape recognition, the patent discusses how pressure can be monitored.

"For example, one or more force sensors underneath or adjacent to the touch-sensitive surface are, optionally, used to measure force at various points on the touch-sensitive surface," it says. "Similarly, a pressure-sensitive tip of a stylus is, optionally, used to determine a pressure of the stylus on the touch-sensitive surface."

The two inventors listed on the patent are Daniel E. Gobera Rubalcava and Ryan S Dixon.

Dixon has very many previous patents, including several that are relevant to this one, such as "Managing real-time handwriting recognition," US Patent No. 10,346,035.

It follows prior Apple patents on the topic including one from 2011 about selective touch recognition, and a 2015 filing about continuous handwriting input.