Editing Handwriting recognition

{{short description|Ability of a computer to receive and interpret intelligible handwritten input}}
{{Use dmy dates|date=January 2023}}
{{About|recognizing the specific letters and words in hand-written text|recognizing the specific person who wrote hand-written text|Graphanalysis}}

[[File:Signiture of country stat, Tex Williams- 2013-04-07 11-37.jpg|thumbnail|Signature of country star Tex Williams]]

'''Handwriting recognition''' ('''HWR'''), also known as '''handwritten text recognition''' ('''HTR'''), is the ability of a computer to receive and interpret intelligible [[handwriting|handwritten]] input from sources such as [[paper]] documents, [[photograph]]s, [[touch-screen]]s and other devices.<ref>{{Cite book |last=Förstner |first=Wolfgang |url=https://www.worldcat.org/oclc/913706869 |title=Mustererkennung 1999 : 21. DAGM-Symposium Bonn, 15.-17. September 1999 |date=1999 |others=Joachim M. Buhmann, Annett Faber, Petko Faber |isbn=978-3-642-60243-6 |location=Berlin, Heidelberg |oclc=913706869 |publisher=Springer Berlin Heidelberg}}</ref><ref>{{Cite book |last=Schenk |first=Joachim |url=https://www.worldcat.org/oclc/609418875 |title=Mensch-maschine-kommunikation : grundlagen von sprach- und bildbasierten benutzerschnittstellen |date=2010 |publisher=Springer |others=Gerhard Rigoll |isbn=978-3-642-05457-0 |location=Heidelberg |oclc=609418875}}</ref> The image of the written text may be sensed "off line" from a piece of paper by optical scanning ([[optical character recognition]]) or [[intelligent word recognition]]. Alternatively, the movements of the pen tip may be sensed "on line", for example by a pen-based computer screen surface, a generally easier task as there are more clues available. A handwriting recognition system handles formatting, performs correct [[Segment (handwriting)|segmentation]] into characters, and finds the most possible words.

== Offline recognition ==
Offline handwriting recognition involves the automatic conversion of text in an image into letter codes that are usable within computer and text-processing applications. The data obtained by this form is regarded as a static representation of handwriting. Offline handwriting recognition is comparatively difficult, as different people have different handwriting styles. And, as of today, OCR engines are primarily focused on machine printed text and [[Intelligent character recognition|ICR]] for hand "printed" (written in capital letters) text.

=== Traditional techniques ===

==== Character extraction ====
Offline character recognition often involves scanning a form or document. This means the individual characters contained in the scanned image will need to be extracted. Tools exist that are capable of performing this step.<ref>[https://sourceforge.net/projects/javaocr/ Java OCR, 5 June 2010]. Retrieved 5 June 2010</ref> However, there are several common imperfections in this step. The most common is when characters that are connected are returned as a single sub-image containing both characters. This causes a major problem in the recognition stage. Yet many algorithms are available that reduce the risk of connected characters.

==== Character recognition ====
After individual characters have been extracted, a recognition engine is used to identify the corresponding computer character. Several different recognition techniques are currently available.

===== Feature extraction =====
[[Feature extraction]] works in a similar fashion to neural network recognizers. However, programmers must manually determine the properties they feel are important. This approach gives the recognizer more control over the properties used in identification. Yet any system using this approach requires substantially more development time than a neural network because the properties are not learned automatically.

=== Modern techniques ===

Where traditional techniques focus on [[image segmentation|segmenting]] individual characters for recognition, modern techniques focus on recognizing all the characters in a segmented line of text. Particularly they focus on [[machine learning]] techniques that are able to learn visual features, avoiding the limiting feature engineering previously used. State-of-the-art methods use [[Convolutional neural network|convolutional networks]] to extract visual features over several overlapping windows of a text line image which a [[recurrent neural network]] uses to produce character probabilities.<ref>Puigcerver, Joan. "Are Multidimensional Recurrent Layers Really Necessary for Handwritten Text Recognition?." Document Analysis and Recognition (ICDAR), 2017 14th IAPR International Conference on. Vol. 1. IEEE, 2017.</ref>

==Online recognition ==
Online handwriting recognition involves the automatic conversion of text as it is written on a special [[digitizer]] or [[Personal digital assistant|PDA]], where a sensor picks up the pen-tip movements as well as pen-up/pen-down switching. This kind of data is known as digital ink and can be regarded as a digital representation of handwriting. The obtained signal is converted into letter codes that are usable within computer and text-processing applications.

The elements of an online handwriting recognition interface typically include:

* a pen or stylus for the user to write with
* a touch sensitive surface, which may be integrated with, or adjacent to, an output display.
* a software application which interprets the movements of the stylus across the writing surface, translating the resulting strokes into digital text.

The process of online handwriting recognition can be broken down into a few general steps:

* preprocessing, 
* feature extraction and 
* classification

The purpose of preprocessing is to discard irrelevant information in the input data, that can negatively affect the recognition.<ref>Huang, B.; Zhang, Y. and Kechadi, M.; ''Preprocessing Techniques for Online Handwriting Recognition. Intelligent Text Categorization and Clustering'', Springer Berlin Heidelberg, 2009, Vol. 164, "Studies in Computational Intelligence" pp. 25–45.</ref> This concerns speed and accuracy. Preprocessing usually consists of binarization, normalization, sampling, smoothing and denoising.<ref>Holzinger, A.; Stocker, C.; Peischl, B. and Simonic, K.-M.; ''[http://www.mdpi.com/1099-4300/14/11/2324 On Using Entropy for Enhancing Handwriting Preprocessing]'', Entropy 2012, 14, pp. 2324–2350.</ref> The second step is feature extraction. Out of the two- or higher-dimensional vector field received from the preprocessing algorithms, higher-dimensional data is extracted. The purpose of this step is to highlight important information for the recognition model. This data may include information like pen pressure, velocity or the changes of writing direction. The last big step is classification. In this step, various models are used to map the extracted features to different classes and thus identifying the characters or words the features represent.

=== Hardware ===
Commercial products incorporating handwriting recognition as a replacement for keyboard input were introduced in the early 1980s. Examples include handwriting terminals such as the [[Pencept]] Penpad<ref>{{Citation
  | title = Pencept Penpad (TM) 200 Product Literature
  | publisher= Pencept, Inc.
  | date=1982-08-15
  | url=http://users.erols.com/rwservices/pens/biblio83.html#Pencept83
  }}</ref> 
and the Inforite point-of-sale terminal.<ref>{{Citation
  | title = Inforite Hand Character Recognition Terminal
  | publisher= Cadre Systems Limited, England
  | date=1982-08-15
  | url=http://users.erols.com/rwservices/pens/biblio83.html#Inforite82
  }}</ref>
With the advent of the large consumer market for personal computers, several commercial products were introduced to replace the keyboard and mouse on a personal computer with a single pointing/handwriting system, such as those from Pencept,<ref name="users.erols.com">{{Citation
  | title = Users Manual for Penpad 320
  | publisher= Pencept, Inc.
  | date=1984-06-15
  | url=http://users.erols.com/rwservices/pens/biblio85.html#Pencept84d
  }}</ref> CIC<ref name="rwservices.no-ip.info">{{Citation
  | title = Handwriter (R) GrafText (TM) System Model GT-5000
  | publisher= Communication Intelligence Corporation
  | date=1985-01-15
  | url=http://users.erols.com/rwservices/pens/biblio85.html#CIC85
  }}</ref> and others.
The first commercially available tablet-type portable computer was the [[Linus Write-Top|Write-Top]] from Linus Technologies, released in July 1988. Its operating system was based on [[MS-DOS]].<ref name=computer>{{cite book | last=Atkinson | first=Paul | date=2010 | url=https://books.google.com/books?id=D5H_OsxEywwC | title=Computer | publisher=Reaktion Books | pages=115–116 | isbn=9781861897374 | via=Google Books}}</ref><ref>{{cite book | last=Delbourg-Delphis | first=Marylène | date=2024 | url=https://books.google.com/books?id=g2z8EAAAQBAJ | title=Beyond Eureka!: The Rocky Roads to Innovating | publisher=Georgetown University Press | page=168 | isbn=9781647124229 | via=Google Books}}</ref>

In the early 1990s, hardware makers including [[NCR Corporation|NCR]], [[IBM]] and [[EO Personal Communicator|EO]] released [[tablet computer]]s running the [[PenPoint OS|PenPoint]] operating system developed by [[GO Corp.]] PenPoint used handwriting recognition and gestures throughout and provided the facilities to third-party software. IBM's tablet computer was the first to use the [[ThinkPad]] name and used IBM's handwriting recognition.  This recognition system was later ported to Microsoft [[Windows for Pen Computing]], and IBM's [[Pen for OS/2]].  None of these were commercially successful.

Advancements in electronics allowed the computing power necessary for handwriting recognition to fit into a smaller form factor than tablet computers, and handwriting recognition is often used as an input method for hand-held [[Personal Digital Assistant|PDA]]s. The first PDA to provide written input was the [[Apple Newton]], which exposed the public to the advantage of a streamlined user interface. However, the device was not a commercial success, owing to the unreliability of the software, which tried to learn a user's writing patterns.  By the time of the release of the [[Newton OS]] 2.0, wherein the handwriting recognition was greatly improved, including unique features still not found in current recognition systems such as modeless error correction, the largely negative first impression had been made. After discontinuation of [[Apple Newton]], the feature was incorporated in Mac OS X 10.2 and later as [[Inkwell (Macintosh)|Inkwell]].

[[Palm, Inc.|Palm]] later launched a successful series of [[Personal Digital Assistant|PDA]]s based on the [[Graffiti (Palm OS)|Graffiti]] recognition system. Graffiti improved usability by defining a set of "unistrokes", or one-stroke forms, for each character. This narrowed the possibility for erroneous input, although memorization of the stroke patterns did increase the learning curve for the user.  The Graffiti handwriting recognition was found to infringe on a patent held by Xerox, and Palm replaced Graffiti with a licensed version of the CIC handwriting recognition which, while also supporting unistroke forms, pre-dated the Xerox patent.  The court finding of infringement was reversed on appeal, and then reversed again on a later appeal.  The parties involved subsequently negotiated a settlement concerning this and other patents.

A [[Tablet computer|Tablet PC]] is a notebook computer with a [[Graphics tablet|digitizer tablet]] and a stylus, which allows a user to handwrite text on the unit's screen. The operating system recognizes the handwriting and converts it into text. [[Windows Vista]] and [[Windows 7]] include personalization features that learn a user's writing patterns or vocabulary for English, Japanese, Chinese Traditional, Chinese Simplified and Korean. The features include a "personalization wizard" that prompts for samples of a user's handwriting and uses them to retrain the system for higher accuracy recognition. This system is distinct from the less advanced handwriting recognition system employed in its [[Windows Mobile]] OS for PDAs.

Although handwriting recognition is an input form that the public has become accustomed to, it has not achieved widespread use in either desktop computers or laptops. It is still generally accepted that [[Alphanumeric keyboard|keyboard]] input is both faster and more reliable.  {{As of|2006}}, many PDAs offer handwriting input, sometimes even accepting natural cursive handwriting, but accuracy is still a problem, and some people still find even a simple [[virtual keyboard|on-screen keyboard]] more efficient.

===Software===
Early software could understand print handwriting where the characters were separated; however, cursive handwriting with connected characters presented [[Sayre's paradox|Sayre's Paradox]], a difficulty involving character segmentation. In 1962 [[Guberman Shelia (Shelija)|Shelia Guberman]], then in Moscow, wrote the first applied pattern recognition program.<ref>Guberman is the inventor of the handwriting recognition technology used today by Microsoft in Windows CE. Source: [https://www.iqt.org/in-q-tel-invests-in-pixlogic/ In-Q-Tel communication, June 3, 2003]</ref> Commercial examples came from companies such as Communications Intelligence Corporation and IBM.

In the early 1990s, two companies&nbsp;&ndash; ParaGraph International and Lexicus&nbsp;&ndash; came up with systems that could understand cursive handwriting recognition. ParaGraph was based in Russia and founded by computer scientist [[Stepan Pachikov]] while Lexicus was founded by [[Ronjon Nag]] and Chris Kortge who were students at Stanford University. The ParaGraph CalliGrapher system was deployed in the Apple Newton systems, and Lexicus Longhand system was made available commercially for the PenPoint and Windows operating system. Lexicus was acquired by Motorola in 1993 and went on to develop Chinese handwriting recognition and [[predictive text]] systems for Motorola. ParaGraph was acquired in 1997 by SGI and its handwriting recognition team formed a P&I division, later acquired from SGI by [[Vadem]]. Microsoft has acquired CalliGrapher handwriting recognition and other digital ink technologies developed by P&I from Vadem in 1999.

Wolfram Mathematica (8.0 or later) also provides a handwriting or text recognition function TextRecognize.

==Research==
[[File:Address Recognition.png|thumbnail|Method used for exploiting contextual information in the first [[handwritten address interpretation]] system developed by [[Sargur Srihari]] and Jonathan Hull<ref name="Integration of handwritten recognition" /> ]]
Handwriting recognition has an active community of academics studying it.  The biggest conferences for handwriting recognition are the International Conference on Frontiers in Handwriting Recognition (ICFHR), held in even-numbered years, and the [[International Conference on Document Analysis and Recognition]] (ICDAR), held in odd-numbered years.  Both of these conferences are endorsed by the IEEE and [[IAPR]].  
In 2021, the ICDAR proceedings will be published by [[Lecture Notes in Computer Science|LNCS]], Springer.

Active areas of research include:
* Online recognition
* Offline recognition
* Signature verification
* [[Handwritten Address Interpretation|Postal address interpretation]]
* Bank-Check processing
* [[Writer Recognition|Writer recognition]]

==Results since  2009==

Since 2009, the [[recurrent neural network]]s and deep [[feedforward neural network|feedforward]] neural networks developed in the research group of [[Jürgen Schmidhuber]] at the [[IDSIA|Swiss AI Lab IDSIA]] have won several international handwriting competitions.<ref>[http://www.kurzweilai.net/how-bio-inspired-deep-learning-keeps-winning-competitions 2012 Kurzweil AI Interview] {{Webarchive|url=https://web.archive.org/web/20180831075249/http://www.kurzweilai.net/how-bio-inspired-deep-learning-keeps-winning-competitions |date=31 August 2018 }} with [[Jürgen Schmidhuber]] on the eight competitions won by his Deep Learning team 2009-2012</ref> In particular, the bi-directional and [[multi-dimensional]] [[Long short-term memory]] (LSTM)<ref>Graves, Alex; and Schmidhuber, Jürgen; ''Offline Handwriting Recognition with Multidimensional Recurrent Neural Networks'', in Bengio, Yoshua; Schuurmans, Dale; Lafferty, John; Williams, Chris K. I.; and Culotta, Aron (eds.), ''Advances in Neural Information Processing Systems 22 (NIPS'22), December 7th–10th, 2009, Vancouver, BC'', Neural Information Processing Systems (NIPS) Foundation, 2009, pp. 545–552</ref><ref>A. Graves, M. Liwicki, S. Fernandez, R. Bertolami, H. Bunke, [[Jürgen Schmidhuber|J. Schmidhuber]]. A Novel Connectionist System for Improved Unconstrained Handwriting Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 5, 2009.</ref> of Alex Graves et al. won three competitions in connected handwriting recognition at the 2009 International Conference on Document Analysis and Recognition (ICDAR), without any prior knowledge about the three different languages (French, Arabic, [[Persian language|Persian]]) to be learned.  Recent [[GPU]]-based [[deep learning]] methods for feedforward networks by Dan Ciresan and colleagues at [[IDSIA]] won the ICDAR 2011 offline Chinese handwriting recognition contest; their neural networks also were the first artificial pattern recognizers to achieve human-competitive performance<ref>D. C. Ciresan, U. Meier, [[Jürgen Schmidhuber|J. Schmidhuber]]. Multi-column Deep Neural Networks for Image Classification. IEEE Conf. on Computer Vision and Pattern Recognition CVPR 2012.</ref> on the famous [[MNIST database|MNIST]] handwritten digits problem<ref>[[Yann LeCun|LeCun, Y.]], Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proc. IEEE, 86, pp. 2278–2324.</ref> of [[Yann LeCun]] and colleagues at [[NYU]].

Benjamin Graham of the [[University of Warwick]] won a 2013 Chinese handwriting recognition contest, with only a 2.61% error rate, by using an approach to [[convolutional neural networks]] that evolved (by 2017) into "sparse convolutional neural networks".<ref>{{cite news |title=Sparse Networks Come to the Aid of Big Physics |url=https://www.quantamagazine.org/sparse-neural-networks-point-physicists-to-useful-data-20230608/ |access-date=17 June 2023 |work=[[Quanta Magazine]] |date=June 2023}}</ref><ref>Graham, Benjamin. "Spatially-sparse convolutional neural networks." arXiv preprint arXiv:1409.6070 (2014).</ref>

== See also ==
* [[AI effect]]
* [[Applications of artificial intelligence]]
* [[Electronic signature]]
* [[eScriptorium]]
* [[Handwriting movement analysis]]
* [[Intelligent character recognition]]
* [[Live Ink Character Recognition Solution]]
* [[Neocognitron]]
* [[Optical character recognition]]
* [[Pen computing]]
* [[Sketch recognition]]
* [[Stylus (computing)]]
* [[Tablet computer|Tablet PC]]

=== Lists ===
* [[Outline of artificial intelligence]]
* [[List of emerging technologies]]

==References==
<!--<nowiki>
See http://en.wikipedia.org/wiki/Wikipedia:Footnotes for an explanation of how to generate footnotes using the <ref> and </ref> tags, and the template below. 
</nowiki>-->
{{reflist|2|refs=<ref name="Integration of handwritten recognition">S. N. Srihari and E. J. Keubert, [http://portal.acm.org/citation.cfm?id=685138 "Integration of handwritten address interpretation technology into the United States Postal Service Remote Computer Reader System"] Proc. Int. Conf. Document Analysis and Recognition (ICDAR) 1997, IEEE-CS Press, pp. 892–896</ref> }}

==External links==
*[http://ruetersward.com/biblio.html Annotated bibliography of references to gesture and pen computing]
*[https://www.youtube.com/watch?v=4xnqKdWMa_8 Notes on the History of Pen-based Computing] – video on [[YouTube]]

{{Pens}}
{{Artificial intelligence navbox}}

{{DEFAULTSORT:Handwriting Recognition}}
[[Category:Handwriting recognition| ]]
[[Category:Pointing-device text input]]
[[Category:Machine learning task]]
[[Category:Computational linguistics]]