Editing Lexical analysis (section)

== Phrase structure ==
Lexical analysis mainly segments the input stream of characters into tokens, simply grouping the characters into pieces and categorizing them. However, the lexing may be significantly more complex; most simply, lexers may omit tokens or insert added tokens. Omitting tokens, notably whitespace and comments, is very common when these are not needed by the compiler. Less commonly, added tokens may be inserted. This is done mainly to group tokens into [[Statement (computer science)|statements]], or statements into blocks, to simplify the parser.

=== Line continuation ===
[[Line continuation]] is a feature of some languages where a newline is normally a statement terminator. Most often, ending a line with a backslash (immediately followed by a [[newline]]) results in the line being ''continued'' – the following line is ''joined'' to the prior line. This is generally done in the lexer: The backslash and newline are discarded, rather than the newline being tokenized. Examples include [[Bash (Unix shell)|bash]],<ref>''[https://www.gnu.org/software/bash/manual/bashref.html Bash Reference Manual]'', [https://www.gnu.org/software/bash/manual/bashref.html#Escape-Character 3.1.2.1 Escape Character]</ref> other shell scripts and Python.<ref name="3.6.4 Documentation">{{cite web|url=https://docs.python.org/|title=3.6.4 Documentation|website=docs.python.org}}</ref>

=== Semicolon insertion ===
Many languages use the semicolon as a statement terminator. Most often this is mandatory, but in some languages the semicolon is optional in many contexts. This is mainly done at the lexer level, where the lexer outputs a semicolon into the token stream, despite one not being present in the input character stream, and is termed ''semicolon insertion'' or ''automatic semicolon insertion''. In these cases, semicolons are part of the formal phrase grammar of the language, but may not be found in input text, as they can be inserted by the lexer. Optional semicolons or other terminators or separators are also sometimes handled at the parser level, notably in the case of [[trailing comma]]s or semicolons.

Semicolon insertion is a feature of [[BCPL]] and its distant descendant [[Go (programming language)|Go]],<ref>''[http://golang.org/doc/effective_go.html Effective Go]'', "[http://golang.org/doc/effective_go.html#semicolons Semicolons]"</ref> though it is absent in B or C.<ref>"[https://groups.google.com/forum/#!topic/golang-nuts/XuMrWI0Q8uk Semicolons in Go]", golang-nuts, Rob 'Commander' Pike, 12/10/09</ref> Semicolon insertion is present in [[JavaScript]], though the rules are somewhat complex and much-criticized; to avoid bugs, some recommend always using semicolons, while others use initial semicolons, termed [[defensive semicolon]]s, at the start of potentially ambiguous statements.

Semicolon insertion (in languages with semicolon-terminated statements) and line continuation (in languages with newline-terminated statements) can be seen as complementary: Semicolon insertion adds a token even though newlines generally do ''not'' generate tokens, while line continuation prevents a token from being generated even though newlines generally ''do'' generate tokens.

=== Off-side rule ===
{{further|Off-side rule}}
The [[off-side rule]] (blocks determined by indenting) can be implemented in the lexer, as in [[Python (programming language)|Python]], where increasing the indenting results in the lexer emitting an INDENT token and decreasing the indenting results in the lexer emitting one or more DEDENT tokens.<ref>{{cite web |title=Lexical analysis > Indentation |url=https://docs.python.org/3.11/reference/lexical_analysis.html#indentation |website=The Python Language Reference |access-date=21 June 2023}}</ref> These tokens correspond to the opening brace <code>{</code> and closing brace <code>}</code> in languages that use braces for blocks and means that the phrase grammar does not depend on whether braces or indenting are used. This requires that the lexer hold state, namely a stack of indent levels, and thus can detect changes in indenting when this changes, and thus the lexical grammar is not [[Context-free grammar|context-free]]: INDENT–DEDENT depend on the contextual information of prior indent levels.