Editing Byte order mark (section)

===UTF-8===
The [[UTF-8]] representation of the BOM is the ([[hexadecimal]]) byte sequence <code>EF&nbsp;BB&nbsp;BF</code>.

The Unicode Standard permits the BOM in [[UTF-8]],<ref>{{cite web|url=https://www.unicode.org/versions/Unicode5.0.0/ch02.pdf |title=The Unicode Standard 5.0, Chapter 2:General Structure |access-date=29 March 2009 |page=36 | quote=Table 2-4. The Seven Unicode Encoding Schemes}}</ref> but does not require or recommend its use.<ref>{{cite web|url=https://www.unicode.org/versions/Unicode5.0.0/ch02.pdf |title=The Unicode Standard 5.0, Chapter 2:General Structure |access-date=30 November 2008 |page=36 |quote=Use of a BOM is neither required nor recommended for UTF-8, but may be encountered in contexts where UTF-8 data is converted from other encoding forms that use a BOM or where the BOM is used as a UTF-8 signature}}</ref> UTF-8 always has the same byte order,<ref name="utf-8-bom">{{cite web|title = FAQ - UTF-8, UTF-16, UTF-32 & BOM: Can a UTF-8 data stream contain the BOM character (in UTF-8 form)? If yes, then can I still assume the remaining UTF-8 bytes are in big-endian order?|url = http://unicode.org/faq/utf_bom.html#bom5|website=Unicode.org|access-date = 4 January 2009}}</ref> so its only use in UTF-8 is to signal at the start that the text stream is encoded in UTF-8, or that it was converted to UTF-8 from a stream that contained an optional BOM. The standard also does not recommend removing a BOM when it is there, so that round-tripping between encodings does not lose information, and so that code that relies on it continues to work.<ref>{{cite web|title=Re: pre-HTML5 and the BOM from Asmus Freytag on 2012-07-13 (Unicode Mail List Archive)|url=https://www.unicode.org/mail-arch/unicode-ml/y2012-m07/0268.html|website=Unicode.org|access-date = 14 July 2012}}</ref><ref>{{cite web|url=https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6378911 |title=Bug ID: JDK-6378911 UTF-8 decoder handling of byte-order mark has changed |website=Bugs.java.com |access-date=14 October 2021}}</ref> The IETF recommends that if a protocol either (a) always uses UTF-8, or (b) has some other way to indicate what encoding is being used, then it "SHOULD forbid use of U+FEFF as a signature."<ref name=rfc3629>{{cite IETF|title = UTF-8, a transformation format of ISO 10646|rfc = 3629|last1 = Yergeau|first1 = Francois|date=November 2003|publisher = [[Internet Engineering Task Force|IETF]]|access-date = 15 May 2014}}</ref> An example of not following this recommendation is the IETF [[Syslog]] protocol which requires text to be in UTF-8 and also requires the BOM.<ref>{{cite IETF |title=The Syslog Protocol |rfc=5424 |sectionname=MSG |section=6.4 |last= Gerhards |first= Rainer |date=March 2009 |publisher=[[Internet Engineering Task Force|IETF]] }}</ref>

Not using a BOM allows text to be backwards-compatible with software designed for [[extended ASCII]]. For instance many programming languages permit non-[[ASCII]] bytes in [[string literal]]s but not at the start of the file.

A BOM is unnecessary for detecting UTF-8 encoding.{{Citation needed|date=May 2024}} UTF-8 is a sparse encoding: a large fraction of possible byte combinations do not result in valid UTF-8 text. Binary data and text in any other encoding are likely to contain byte sequences that are invalid as UTF-8, so existence of such invalid sequences indicates the file is not UTF-8, while lack of invalid sequences is a very strong indication the text ''is'' UTF-8. Practically the only exception is text containing only ASCII-range bytes, as this may be a non-ASCII 7-bit encoding, but this is unlikely in any modern data and even then the difference from ASCII is minor (such as changing '\' to '¥').

[[Microsoft]] compilers<ref>{{cite web|url=http://alfps.wordpress.com/2011/11/22/unicode-part-1-windows-console-io-approaches/|title=Unicode part 1: Windows console i/o approaches|author=Alf P. Steinbach|access-date=24 March 2012|year=2011|quote=However, since the C++ source code was encoded as UTF-8 without BOM (as is usual in Linux), the Visual C++ compiler erroneously assumed that the source code was encoded as Windows ANSI.}}</ref> and interpreters, and many pieces of software on [[Microsoft Windows]] such as [[Notepad (Windows)|Notepad]] (prior to Windows 10 Build 1903<ref>{{Cite web |title=Windows 10 Notepad is Getting Better UTF-8 Encoding Support |url=https://www.bleepingcomputer.com/news/microsoft/windows-10-notepad-is-getting-better-utf-8-encoding-support/ |access-date=2023-03-07 |website=BleepingComputer |language=en-us}}</ref>) treat the BOM as a required [[Magic number (programming)#Magic numbers in files|magic number]] rather than use heuristics. These tools add a BOM when saving text as UTF-8, and cannot interpret UTF-8 unless the BOM is present or the file contains only ASCII. [[Windows PowerShell]] (up to 5.1) will add a BOM when it saves UTF-8 XML documents. However, PowerShell Core 6 has added a <code>-Encoding</code> switch on some cmdlets called utf8NoBOM so that document can be saved without BOM. [[Google Docs]] also adds a BOM when converting a document to a [[plain text]] file for download.