Editing Digital image processing (section)

==History==
{{See|Digital image#History|Digital imaging#History}}

Many of the techniques of [[digital image]] processing, or digital picture processing as it often was called, were developed in the 1960s, at [[Bell Laboratories]], the [[Jet Propulsion Laboratory]], [[Massachusetts Institute of Technology]], [[University of Maryland, College Park|University of Maryland]], and a few other research facilities, with application to [[satellite imagery]], [[Wirephoto|wire-photo]] standards conversion, [[medical physics|medical imaging]], [[videophone]], [[character recognition]], and photograph enhancement.<ref>Azriel Rosenfeld, ''Picture Processing by Computer'', New York: Academic Press, 1969</ref> The purpose of early image processing was to improve the quality of the image. It was aimed for human beings to improve the visual effect of people. In image processing, the input is a low-quality image, and the output is an image with improved quality. Common image processing include image enhancement, restoration, encoding, and compression. The first successful application was the American Jet Propulsion Laboratory (JPL). They used image processing techniques such as geometric correction, gradation transformation, noise removal, etc. on the thousands of lunar photos sent back by the Space Detector Ranger 7 in 1964, taking into account the position of the Sun and the environment of the Moon. The impact of the successful mapping of the Moon's surface map by the computer has been a success. Later, more complex image processing was performed on the nearly 100,000 photos sent back by the spacecraft, so that the topographic map, color map and panoramic mosaic of the Moon were obtained, which achieved extraordinary results and laid a solid foundation for human landing on the Moon.<ref name=":1">{{Cite book|title=Digital image processing|last=Gonzalez, Rafael C.|date=2008|publisher=Prentice Hall|others=Woods, Richard E. (Richard Eugene), 1954–|isbn=978-0-13-168728-8|edition= 3rd|location=Upper Saddle River, N.J.|pages=23–28|oclc=137312858}}</ref>

The cost of processing was fairly high, however, with the computing equipment of that era. That changed in the 1970s, when digital image processing proliferated as cheaper computers and dedicated hardware became available. This led to images being processed in real-time, for some dedicated problems such as [[television standards conversion]]. As [[general-purpose computer]]s became faster, they started to take over the role of dedicated hardware for all but the most specialized and computer-intensive operations. With the fast computers and signal processors available in the 2000s, digital image processing has become the most common form of image processing, and is generally used because it is not only the most versatile method, but also the cheapest.

===Image sensors===
{{Main|Image sensor}}

The basis for modern [[image sensors]] is [[metal–oxide–semiconductor]] (MOS) technology,<ref name="Williams">{{cite book |last1=Williams |first1=J. B. |title=The Electronics Revolution: Inventing the Future |date=2017 |publisher=Springer |isbn=978-3-319-49088-5 |pages=245–8 |url=https://books.google.com/books?id=v4QlDwAAQBAJ&pg=PA245}}</ref> invented at Bell Labs between 1955 and 1960,<ref>{{Cite patent|number=US2802760A|title=Oxidation of semiconductive surfaces for controlled diffusion|gdate=1957-08-13|invent1=Lincoln|invent2=Frosch|inventor1-first=Derick|inventor2-first=Carl J.|url=https://patents.google.com/patent/US2802760A}}</ref><ref>{{Cite journal |last1=Frosch |first1=C. J. |last2=Derick |first2=L |date=1957 |title=Surface Protection and Selective Masking during Diffusion in Silicon |url=https://iopscience.iop.org/article/10.1149/1.2428650 |journal=Journal of the Electrochemical Society |language=en |volume=104 |issue=9 |pages=547 |doi=10.1149/1.2428650|url-access=subscription }}</ref><ref>{{Cite journal |last=KAHNG |first=D. |date=1961 |title=Silicon-Silicon Dioxide Surface Device |url=https://doi.org/10.1142/9789814503464_0076 |journal=Technical Memorandum of Bell Laboratories |pages=583–596 |doi=10.1142/9789814503464_0076 |isbn=978-981-02-0209-5|url-access=subscription }}</ref><ref>{{Cite book |last=Lojek |first=Bo |title=History of Semiconductor Engineering |date=2007 |publisher=Springer-Verlag Berlin Heidelberg |isbn=978-3-540-34258-8 |location=Berlin, Heidelberg |page=321}}</ref><ref>{{Cite journal |last1=Ligenza |first1=J.R. |last2=Spitzer |first2=W.G. |date=1960 |title=The mechanisms for silicon oxidation in steam and oxygen |url=https://linkinghub.elsevier.com/retrieve/pii/0022369760902195 |journal=Journal of Physics and Chemistry of Solids |language=en |volume=14 |pages=131–136 |bibcode=1960JPCS...14..131L |doi=10.1016/0022-3697(60)90219-5|url-access=subscription }}</ref><ref name="Lojek1202">{{cite book |last1=Lojek |first1=Bo |title=History of Semiconductor Engineering |date=2007 |publisher=[[Springer Science & Business Media]] |isbn=9783540342588 |page=120}}</ref> This led to the development of digital [[semiconductor]] image sensors, including the [[charge-coupled device]] (CCD) and later the [[CMOS sensor]].<ref name="Williams"/>

The charge-coupled device was invented by [[Willard S. Boyle]] and [[George E. Smith]] at Bell Labs in 1969.<ref>{{Cite book | title = Scientific charge-coupled devices | author = James R. Janesick | publisher = SPIE Press | year = 2001 | isbn = 978-0-8194-3698-6 | pages = 3–4 | url = https://books.google.com/books?id=3GyE4SWytn4C&pg=PA3 }}</ref> While researching MOS technology, they realized that an electric charge was the analogy of the magnetic bubble and that it could be stored on a tiny [[MOS capacitor]]. As it was fairly straightforward to [[semiconductor device fabrication|fabricate]] a series of MOS capacitors in a row, they connected a suitable voltage to them so that the charge could be stepped along from one to the next.<ref name="Williams"/> The CCD is a semiconductor circuit that was later used in the first [[digital video camera]]s for [[television broadcasting]].<ref>{{cite journal|last1=Boyle|first1=William S|last2=Smith|first2=George E.|date=1970|title=Charge Coupled Semiconductor Devices|journal=Bell Syst. Tech. J.|volume=49|issue=4|pages=587–593|doi=10.1002/j.1538-7305.1970.tb01790.x|bibcode=1970BSTJ...49..587B }}</ref>

The [[NMOS logic|NMOS]] [[active-pixel sensor]] (APS) was invented by [[Olympus Corporation|Olympus]] in Japan during the mid-1980s. This was enabled by advances in MOS [[semiconductor device fabrication]], with [[MOSFET scaling]] reaching smaller [[List of semiconductor scale examples|micron and then sub-micron]] levels.<ref name=fossum93>{{cite book |last1=Fossum |first1=Eric R. |author1-link=Eric Fossum |title= Charge-Coupled Devices and Solid State Optical Sensors III |series=Proceedings of the SPIE |volume=1900 |date=12 July 1993 |doi=10.1117/12.148585 |pages=2–14 |editor1-last=Blouke |editor1-first=Morley M.|citeseerx=10.1.1.408.6558 |bibcode=1993SPIE.1900....2F |chapter=Active pixel sensors: Are CCDS dinosaurs? |s2cid=10556755 }}</ref><ref>{{cite web |last1=Fossum |first1=Eric R. |s2cid=18831792 |author1-link=Eric Fossum |title=Active Pixel Sensors |website=Eric Fossum |year=2007 |url=http://ericfossum.com/Publications/Papers/Active%20Pixel%20Sensors%20LASER%20FOCUS.pdf |archive-url=https://web.archive.org/web/20190829162855/http://ericfossum.com/Publications/Papers/Active%20Pixel%20Sensors%20LASER%20FOCUS.pdf |archive-date=2019-08-29 |url-status=live}}</ref> The NMOS APS was fabricated by Tsutomu Nakamura's team at Olympus in 1985.<ref>{{cite journal |last1=Matsumoto |first1=Kazuya |last2=Nakamura |first2=Tsutomu |last3=Yusa |first3=Atsushi |last4=Nagai |first4=Shohei |display-authors=1|date=1985 |title=A new MOS phototransistor operating in a non-destructive readout mode |journal=Japanese Journal of Applied Physics |volume=24 |issue=5A |page=L323|doi=10.1143/JJAP.24.L323 |bibcode=1985JaJAP..24L.323M |s2cid=108450116 }}</ref> The [[CMOS]] active-pixel sensor (CMOS sensor) was later developed by [[Eric Fossum]]'s team at the [[NASA]] [[Jet Propulsion Laboratory]] in 1993.<ref name="Fossum2014">{{cite journal |last1=Fossum |first1=Eric R. |author1-link=Eric Fossum |last2=Hondongwa |first2=D. B. |title=A Review of the Pinned Photodiode for CCD and CMOS Image Sensors |journal=IEEE Journal of the Electron Devices Society |date=2014 |volume=2 |issue=3 |pages=33–43 |doi=10.1109/JEDS.2014.2306412 |doi-access=free }}</ref> By 2007, sales of CMOS sensors had surpassed CCD sensors.<ref>{{cite news |title=CMOS Image Sensor Sales Stay on Record-Breaking Pace |url=http://www.icinsights.com/news/bulletins/CMOS-Image-Sensor-Sales-Stay-On-RecordBreaking-Pace/ |access-date=6 October 2019 |work=IC Insights |date=8 May 2018 |archive-url=https://web.archive.org/web/20190621180401/http://www.icinsights.com/news/bulletins/CMOS-Image-Sensor-Sales-Stay-On-RecordBreaking-Pace/ |archive-date=21 June 2019 |url-status=live }}</ref>

MOS image sensors are widely used in [[optical mouse]] technology. The first optical mouse, invented by [[Richard F. Lyon]] at [[Xerox]] in 1980, used a [[6 μm process|5{{nbsp}}μm]] [[NMOS logic|NMOS]] [[integrated circuit]] sensor chip.<ref>{{cite book |last1=Lyon |first1=Richard F. |title=Advances in Embedded Computer Vision |date=2014 |publisher=Springer |isbn=9783319093871 |pages=3–22 (3) |chapter=The Optical Mouse: Early Biomimetic Embedded Vision |author1-link=Richard F. Lyon |chapter-url=https://books.google.com/books?id=p_GbBQAAQBAJ&pg=PA3}}</ref><ref>{{cite book |last1=Lyon |first1=Richard F. |title=VLSI Systems and Computations |date=August 1981 |publisher=Computer Science Press |isbn=978-3-642-68404-3 |editor1=H. T. Kung |pages=1–19 |chapter=The Optical Mouse, and an Architectural Methodology for Smart Digital Sensors |doi=10.1007/978-3-642-68402-9_1 |author1-link=Richard F. Lyon |editor2=Robert F. Sproull |editor3=Guy L. Steele |chapter-url=http://bitsavers.trailing-edge.com/pdf/xerox/parc/techReports/VLSI-81-1_The_Optical_Mouse.pdf |archive-url=https://web.archive.org/web/20140226021235/http://bitsavers.trailing-edge.com/pdf/xerox/parc/techReports/VLSI-81-1_The_Optical_Mouse.pdf |archive-date=2014-02-26 |url-status=live |s2cid=60722329}}</ref> Since the first commercial optical mouse, the [[IntelliMouse]] introduced in 1999, most optical mouse devices use CMOS sensors.<ref>{{cite web |last1=Brain |first1=Marshall |last2=Carmack |first2=Carmen |date=24 April 2000 |title=How Computer Mice Work |url=https://computer.howstuffworks.com/mouse4.htm |access-date=9 October 2019 |website=[[HowStuffWorks]] |language=en}}</ref><ref name="hackaday">{{cite web |last1=Benchoff |first1=Brian |date=17 April 2016 |title=Building the First Digital Camera |url=http://hackaday.com/2016/04/17/building-the-first-digital-camera/ |access-date=30 April 2016 |website=[[Hackaday]] |quote=the Cyclops was the first digital camera}}</ref>

===Image compression===
{{Main|Image compression}}

An important development in digital [[image compression]] technology was the [[discrete cosine transform]] (DCT), a [[lossy compression]] technique first proposed by [[N. Ahmed|Nasir Ahmed]] in 1972.<ref name="Ahmed">{{cite journal |last=Ahmed |first=Nasir |author-link=N. Ahmed |title=How I Came Up With the Discrete Cosine Transform |journal=[[Digital Signal Processing (journal)|Digital Signal Processing]] |date=January 1991 |volume=1 |issue=1 |pages=4–5 |doi=10.1016/1051-2004(91)90086-Z |bibcode=1991DSP.....1....4A |url=https://www.scribd.com/doc/52879771/DCT-History-How-I-Came-Up-with-the-Discrete-Cosine-Transform |access-date=10 October 2019 |archive-url=https://web.archive.org/web/20160610013109/https://www.scribd.com/doc/52879771/DCT-History-How-I-Came-Up-with-the-Discrete-Cosine-Transform |archive-date=10 June 2016 |url-status=live |url-access=subscription }}</ref> DCT compression became the basis for [[JPEG]], which was introduced by the [[Joint Photographic Experts Group]] in 1992.<ref name="t81">{{cite web |title=T.81 – Digital compression and coding of continuous-tone still images – requirements and guidelines |url=https://www.w3.org/Graphics/JPEG/itu-t81.pdf |publisher=[[CCITT]] |date=September 1992 |access-date=12 July 2019 |archive-url=https://web.archive.org/web/20190717052727/http://www.w3.org/Graphics/JPEG/itu-t81.pdf |archive-date=17 July 2019 |url-status=live }}</ref> JPEG compresses images down to much smaller file sizes, and has become the most widely used [[image file format]] on the [[Internet]].<ref>{{cite web |title=The JPEG image format explained |url=https://home.bt.com/tech-gadgets/photography/what-is-a-jpeg-11364206889349 |publisher=[[BT Group]] |first1= Joe |last1=Svetlik  |access-date=5 August 2019 |date=31 May 2018 |archive-url=https://web.archive.org/web/20190805194553/https://home.bt.com/tech-gadgets/photography/what-is-a-jpeg-11364206889349 |archive-date=5 August 2019 |url-status=dead }}</ref> Its highly efficient DCT compression algorithm was largely responsible for the wide proliferation of [[digital images]] and [[digital photo]]s,<ref name="Atlantic">{{cite web |date=24 September 2013 |title=What Is a JPEG? The Invisible Object You See Every Day |url=https://www.theatlantic.com/technology/archive/2013/09/what-is-a-jpeg-the-invisible-object-you-see-every-day/279954/ |first1=Paul |last1=Caplan |url-access=subscription |url-status=live |archive-url=https://web.archive.org/web/20191009054159/https://www.theatlantic.com/technology/archive/2013/09/what-is-a-jpeg-the-invisible-object-you-see-every-day/279954/ |archive-date=9 October 2019 |access-date=13 September 2019 |website=[[The Atlantic]]}}</ref> with several billion JPEG images produced every day {{as of|2015|lc=y}}.<ref>{{cite news |last1=Baraniuk |first1=Chris |title=JPeg lockdown: Restriction options sought by committee  |url=https://www.bbc.co.uk/news/technology-34538705 |access-date=13 September 2019 |publisher=[[BBC News]]|date=15 October 2015 |archive-url=https://web.archive.org/web/20191009193610/https://www.bbc.co.uk/news/technology-34538705 |archive-date=9 October 2019 |url-status=live }}</ref>

Medical imaging techniques produce very large amounts of data, especially from CT, MRI and PET modalities. As a result, storage and communications of electronic image data are prohibitive without the use of compression.<ref>{{Cite journal |last1=Nagornov |first1=Nikolay N. |last2=Lyakhov |first2=Pavel A. |last3=Valueva |first3=Maria V. |last4=Bergerman |first4=Maxim V. |date=2022 |title=RNS-Based FPGA Accelerators for High-Quality 3D Medical Image Wavelet Processing Using Scaled Filter Coefficients |s2cid-access=free |journal=IEEE Access |volume=10 |pages=19215–19231 |doi=10.1109/ACCESS.2022.3151361 |issn=2169-3536 |s2cid=246895876 |quote=Medical imaging systems produce increasingly accurate images with improved quality using higher spatial resolutions and color bit-depth. Such improvements increase the amount of information that needs to be stored, processed, and transmitted. |doi-access=free|bibcode=2022IEEEA..1019215N }}</ref><ref>{{Cite journal |last1=Dhouib |first1=D. |last2=Naït-Ali |first2=A. |last3=Olivier |first3=C. |last4=Naceur |first4=M.S. |date=June 2021 |title=ROI-Based Compression Strategy of 3D MRI Brain Datasets for Wireless Communications |url=https://linkinghub.elsevier.com/retrieve/pii/S1959031820300853 |journal=IRBM |language=en |volume=42 |issue=3 |pages=146–153 |doi=10.1016/j.irbm.2020.05.001 |s2cid=219437400 |quote=Because of the large amount of medical imaging data, the transmission process becomes complicated in telemedicine applications. Thus, in order to adapt the data bit streams to the constraints related to the limitation of the bandwidths a reduction of the size of the data by compression of the images is essential.|url-access=subscription }}</ref> [[JPEG 2000]] image compression is used by the [[DICOM]] standard for storage and transmission of medical images. The cost and feasibility of accessing large image data sets over low or various bandwidths are further addressed by use of another DICOM standard, called [[JPIP]], to enable efficient streaming of the [[JPEG 2000]] compressed image data.<ref>{{Cite journal |last1=Xin |first1=Gangtao |last2=Fan |first2=Pingyi |date=2021-06-11 |title=A lossless compression method for multi-component medical images based on big data mining |journal=Scientific Reports |language=en |volume=11 |issue=1 |pages=12372 |doi=10.1038/s41598-021-91920-x |issn=2045-2322|doi-access=free |pmid=34117350 |pmc=8196061 }}</ref>

===Digital signal processor (DSP)===
{{Main|Digital signal processor}}

Electronic [[signal processing]] was revolutionized by the wide adoption of [[MOS technology]] in the 1970s.<ref name="Grant">{{cite book |last1=Grant |first1=Duncan Andrew |last2=Gowar |first2=John |title=Power MOSFETS: theory and applications |date=1989 |publisher=[[Wiley (publisher)|Wiley]] |isbn=978-0-471-82867-9 |page=1 |url=https://books.google.com/books?id=ZiZTAAAAMAAJ |quote=The metal–oxide–semiconductor field-effect transistor (MOSFET) is the most commonly used active device in the very large-scale integration of digital integrated circuits (VLSI). During the 1970s these components revolutionized electronic signal processing, control systems and computers.}}</ref> [[MOS integrated circuit]] technology was the basis for the first single-chip [[microprocessors]] and [[microcontrollers]] in the early 1970s,<ref name="ieee">{{cite journal |last1=Shirriff |first1=Ken |title=The Surprising Story of the First Microprocessors |journal=[[IEEE Spectrum]] |date=30 August 2016 |volume=53 |issue=9 |pages=48–54 |publisher=[[Institute of Electrical and Electronics Engineers]] |doi=10.1109/MSPEC.2016.7551353 |s2cid=32003640 |url=https://spectrum.ieee.org/the-surprising-story-of-the-first-microprocessors |access-date=13 October 2019 |archive-url=https://web.archive.org/web/20191013012248/https://spectrum.ieee.org/tech-history/silicon-revolution/the-surprising-story-of-the-first-microprocessors |archive-date=13 October 2019 |url-status=live |url-access=subscription }}</ref> and then the first single-chip [[digital signal processor]] (DSP) chips in the late 1970s.<ref name="computerhistory1979">{{cite web |title=1979: Single Chip Digital Signal Processor Introduced |url=https://www.computerhistory.org/siliconengine/single-chip-digital-signal-processor-introduced/ |website=The Silicon Engine |publisher=[[Computer History Museum]] |access-date=14 October 2019 |archive-url=https://web.archive.org/web/20191003072500/https://www.computerhistory.org/siliconengine/single-chip-digital-signal-processor-introduced/ |archive-date=3 October 2019 |url-status=live }}</ref><ref name="Taranovich">{{cite web |last1=Taranovich |first1=Steve |title=30 years of DSP: From a child's toy to 4G and beyond |url=https://www.edn.com/design/systems-design/4394792/30-years-of-DSP--From-a-child-s-toy-to-4G-and-beyond |website=[[EDN (magazine)|EDN]] |access-date=14 October 2019 |date=27 August 2012 |archive-url=https://web.archive.org/web/20191014044347/https://www.edn.com/design/systems-design/4394792/30-years-of-DSP--From-a-child-s-toy-to-4G-and-beyond |archive-date=14 October 2019 |url-status=live }}</ref> DSP chips have since been widely used in digital image processing.<ref name="computerhistory1979"/>

The [[discrete cosine transform]] (DCT) [[image compression]] algorithm has been widely implemented in DSP chips, with many companies developing DSP chips based on DCT technology. DCTs are widely used for [[encoding]], decoding, [[video coding]], [[audio coding]], [[multiplexing]], control signals, [[signaling]], [[analog-to-digital conversion]], formatting [[luminance]] and color differences, and color formats such as [[YUV444]] and [[YUV411]]. DCTs are also used for encoding operations such as [[motion estimation]], [[motion compensation]], [[inter-frame]] prediction, [[Quantization (signal processing)|quantization]], perceptual weighting, [[entropy encoding]], variable encoding, and [[motion vector]]s, and decoding operations such as the inverse operation between different color formats ([[YIQ]], [[YUV]] and [[RGB]]) for display purposes. DCTs are also commonly used for [[high-definition television]] (HDTV) encoder/decoder chips.<ref name="Stankovic">{{cite journal |last1=Stanković |first1=Radomir S. |last2=Astola |first2=Jaakko T. |title=Reminiscences of the Early Work in DCT: Interview with K.R. Rao |journal=Reprints from the Early Days of Information Sciences |date=2012 |volume=60 |url=http://ticsp.cs.tut.fi/reports/ticsp-report-60-reprint-rao-corrected.pdf |access-date=13 October 2019 |archive-url=https://web.archive.org/web/20191013204147/http://ticsp.cs.tut.fi/reports/ticsp-report-60-reprint-rao-corrected.pdf |archive-date=13 October 2019 |url-status=live }}</ref>