Editing Lexical analysis (section)

== Rule-based programs ==
A rule-based program, performing lexical tokenization, is called ''tokenizer'',<ref>{{cite web|url=http://www.cs.man.ac.uk/~pjj/farrell/comp3.html|title=Anatomy of a Compiler and The Tokenizer|website=www.cs.man.ac.uk}}</ref> or ''scanner'', although ''scanner'' is also a term for the first stage of a lexer. A lexer forms the first phase of a [[compiler frontend]] in processing. Analysis generally occurs in one pass. Lexers and parsers are most often used for compilers, but can be used for other computer language tools, such as [[prettyprint]]ers or [[Lint (software)|linters]]. Lexing can be divided into two stages: the ''scanning'', which segments the input string into syntactic units called ''lexemes'' and categorizes these into token classes, and the ''evaluating'', which converts lexemes into processed values.

Lexers are generally quite simple, with most of the complexity deferred to the [[Parsing|syntactic analysis]] or [[Semantic analysis (compilers)|semantic analysis]] phases, and can often be generated by a [[#Lexer generator|lexer generator]], notably [[Lex (software)|lex]] or derivatives. However, lexers can sometimes include some complexity, such as [[#Phrase structure|phrase structure]] processing to make input easier and simplify the parser, and may be written partly or fully by hand, either to support more features or for performance.