Scribble for iPadOS could ask users what they've written down

Apple's introduction of Scribble in iPadOS 14 introduced the ability to perform text entry in text fields by simply writing on the iPad display using an Apple Pencil. The system takes advantage of the massive processing capabilities of the tablet to perform as near to real-time processing as possible.

This has allowed users of various languages to write text, and for the iPad to automatically recognize it and insert the typed version into a field or document.

Given that standard handwriting recognition systems are based on elements such as stroke order for markings of letters, such as the Hidden Markov Method, Apple's system seems to ignore that for a more universal approach. One that can handle multiple languages at a time, and with rapid processing.

In a patent granted by the U.S. Patent and Trademark Office on Tuesday and titled "Managing real-time handwriting recognition," Apple offers a look behind the curtain at how such a feature functions.

The patent mentions the use of a "universal recognizer," which is trained using a "large multi-script corpus of writing samples for characters in different languages and scripts." This learning is independent of language, script, stroke order, and stroke direction.

As a result, this creates a machine learning model that can perform mixed language recognition in a variety of handwriting input methods, without requiring manual switching between languages.

Such a form of training offers some benefits to users, like the ability to recognize script that's written using pen strokes that are in completely different orders to traditional handwriting. It could also be feasible for a system to switch between languages instantly, allowing for words and their translations to be instantly written in native scripts.

While it is trained without needing stroke order, the system still knows this information, which can be used to determine other elements. For example, if a character is ambiguous to a system, the stroke order could help determine which letter a user intends to have written.

The amount of data that could be taken into account could also include variability in speed, the tempo, and stroke continuity, which can all inform the system of the character or word the user wanted to write.

It is also proposed that there is a candidate display area on view, that shows potential words and phrases that the user may be writing. The system could use the main candidate from the list for automatic insertion, but the user could also manually select one of the other terms instead.

This could be similar to existing text tools that automatically complete or correct terms being typed. Apple also proposes that tabs could be used to sort the candidates quickly, and to allow users wider choice, such as access to an emoji menu.

While languages are the primary focus, the patent also allows for users to enter emoji into a document, simply by drawing a representation of it.

The patent also explains that the set of strokes could fade over time, removing itself from the display as the user continues to write. This time is also used to process the writing and to insert the typed text version into the document or text entry box.

Originally filed on July 8, 2019, the patent was invented by Mei-Qun Xia, Jannes G. Dolfing, Ryan S. Dixon, Karl M. Groethe, Karan Misra, Jerome R. Bellegarda, and Ueli Meier.

Apple files numerous patent applications on a weekly basis, but while the existence of a patent indicates areas of interest for its research and development teams, it's not guaranteed any patented ideas will ever appear in a future product or service.

Much of what is mentioned in the patent already exists in the form of iPadOS' Scribble feature. Users write out text using the Apple Pencil and it fades, and is automatically entered into where it's needed.

Currently, Scribble is limited to a small range of languages. Also, users need to manually enable support for different language scripts, rather than Scribble automatically understanding all installed languages.

There are currently no suggestions by Scribble for other words or characters that a user may have written down. For the moment, Scribble places the most likely candidate automatically, forcing users to delete or edit incorrect entries.

Stay on top of all Apple news right from your HomePod. Say, "Hey, Siri, play AppleInsider," and you'll get latest AppleInsider Podcast. Or ask your HomePod mini for "AppleInsider Daily" instead and you'll hear a fast update direct from our news team. And, if you're interested in Apple-centric home automation, say "Hey, Siri, play HomeKit Insider," and you'll be listening to our newest specialized podcast in moments.