Editing Color balance

{{Short description|Adjustment of color intensities in photography}}
{{About|the process applied to still images|the equivalent process applied to video|Color grading}}
[[File:Lily-M7292-As-shot-and-manual.jpg|thumb|right|300px|The left half shows the photo as it came from the digital camera. The right half shows the photo adjusted to make a gray surface neutral in the same light.]]

In [[photography]] and [[image processing]], '''color balance''' is the global adjustment of the intensities of the colors (typically red, green, and blue [[primary colors]]).  An important goal of this adjustment is to render specific colors – particularly neutral colors like white or grey – correctly. Hence, the general method is sometimes called '''gray balance''', '''neutral balance''', or '''white balance'''.  Color balance changes the overall mixture of colors in an image and is used for [[color correction]]. Generalized versions of color balance are used to correct colors other than neutrals or to deliberately change them for effect. ''' White balance''' is one of the most common kinds of balancing, and is when colors are adjusted to make a white object (such as a piece of paper or a wall) appear white and not a shade of any other colour.

Image data acquired by sensors – either [[photographic film|film]] or electronic [[image sensor]]s – must be transformed from the acquired values to new values that are appropriate for color reproduction or display.  Several aspects of the acquisition and display process make such color correction essential – including that the acquisition sensors do not match the sensors in the human eye, that the properties of the display medium must be accounted for, and that the ambient viewing conditions of the acquisition differ from the display viewing conditions.

The color balance operations in popular [[image editing]] applications usually operate directly on the red, green, and blue channel [[pixel]] values,<ref>{{Cite book| title = The Gimp for Linux and Unix | author = Phyllis Davis | publisher = Peachpit Press | year = 2000 | isbn = 978-0-201-70253-8 | url = https://books.google.com/books?id=0sEnoWrMw-gC&q=%22color+balance%22+channels&pg=PA135 | page = 134}}</ref><ref>{{Cite book| title = Adobe Photoshop 6.0 | author = Adobe Creative Team | publisher = Adobe Press | year = 2000 | isbn = 978-0-201-71016-8 | url = https://books.google.com/books?id=MRtx2-0GZc4C&q=%22color+balance%22+channels&pg=PA277 | page = 278 }}{{Request quotation|date=April 2021}}<!-- Did Adobe explicitly spell out their algorithm for white balance, which most likely 
uses CIEXYZ or similar to select an illuminant behind the scenes, or did the book just explain how to balance with the channel mixer?  One of these things sports this statement and the other doesn't. -->  </ref> without respect to any color sensing or reproduction model.  In film photography, color balance is typically achieved by using [[color correction filter]]s over the lights or on the camera lens.<ref>{{Cite book| title = Cinematography: Theory and Practice : Imagemaking for Cinematographers, Directors, and Videographers | author = Blain Brown | publisher = Focal Press | year = 2002 | isbn = 978-0-240-80500-9| url = https://books.google.com/books?id=1JL2jFbNPNAC&q=%22color+balance%22&pg=PA170 | page=170 }}</ref>

==Generalized color balance==
[[File:Color balancing girl.jpg|thumb|right|300px|Example of color balancing]]
Sometimes the adjustment to keep neutrals neutral is called ''white balance'', and the phrase ''color balance'' refers to the adjustment that in addition makes other colors in a displayed image appear to have the same general appearance as the colors in an original scene.<ref>{{Cite book| title = Introduction to Color Imaging Science | url = https://archive.org/details/introductiontoco00leeh_034 | url-access = limited | author = Hsien-Che Lee | publisher = Cambridge University Press | year = 2005 | isbn = 978-0-521-84388-1 | page=[https://archive.org/details/introductiontoco00leeh_034/page/n471 450] }}</ref> It is particularly important that neutral (gray, neutral, white) colors in a scene appear neutral in the reproduction.<ref>[http://www.nikondigital.org/articles/white_balance.htm White Balance]. ''Nikon Digital''. {{retrieved|access-date=October 12, 2016}}</ref>

===Psychological color balance===
Humans relate to [[Human skin color|flesh tones]] more critically than other colors. Trees, grass and sky can all be off without concern, but if human flesh tones are 'off' then the human subject can look sick or dead. To address this critical color balance issue, the tri-color primaries themselves are formulated to ''not'' balance as a true neutral color. The purpose of this color primary imbalance is to more faithfully reproduce the flesh tones through the entire brightness range.

==Illuminant estimation and adaptation==
[[File:Clifton Beach 5.jpg|thumb|right|300px|A seascape photograph at [[Clifton Beach, Tasmania|Clifton Beach]], [[South Arm, Tasmania|South Arm]], [[Tasmania]], Australia. The white balance has been adjusted towards the warm side for creative effect.]]
[[File:ColorChecker100423.jpg|thumb|right|300px|Photograph of a [[ColorChecker]] as a reference shot for color balance adjustments.]]
[[File:Government Center Miami color balance comparison.jpg|thumb|right|300px|Two photos of a high-rise building shot within a minute of each other with an entry-level point-and-shoot camera. Left photo shows a "normal", more accurate color balance, while the right side shows a "vivid" color balance, in-camera effects and no post-production besides black background.]]
[[File:PIA16800-MarsCuriosityRover-MtSharp-ColorVersions-20120823.jpg|thumb|right|300px|Comparison of color versions (raw, natural, white balance) of [[Mount Sharp|Mount Sharp (Aeolis Mons)]] on [[Mars]]]]
[[File:PIA16068 - Mars Curiosity Rover - Aeolis Mons - 20120817.jpg|thumb|right|300px|A white-balanced image of Mount Sharp (Aeolis Mons) on Mars]]
Most digital cameras have means to select color correction based on the type of scene lighting, using either manual lighting selection, automatic white balance, or custom white balance.<ref>{{Cite book|last1=Afifi|first1=Mahmoud|last2=Price|first2=Brian|last3=Cohen|first3=Scott|last4=Brown|first4=Michael S|title=2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) |chapter=When Color Constancy Goes Wrong: Correcting Improperly White-Balanced Images |date=2019|chapter-url=http://openaccess.thecvf.com/content_CVPR_2019/papers/Afifi_When_Color_Constancy_Goes_Wrong_Correcting_Improperly_White-Balanced_Images_CVPR_2019_paper.pdf|pages=1535–1544|doi=10.1109/cvpr.2019.00163|isbn=978-1-7281-3293-8|s2cid=196195956}}</ref> The algorithms for these processes perform generalized [[chromatic adaptation]].

Many methods exist for color balancing.  Setting a button on a camera is a way for the user to indicate to the processor the nature of the scene lighting. Another option on some cameras is a button which one may press when the camera is pointed at a [[gray card]] or other neutral colored object. This captures an image of the ambient light, which enables a digital camera to set the correct color balance for that light.

There is a large literature on how one might estimate the ambient lighting from the camera data and then use this information to transform the image data. A variety of algorithms have been proposed, and the quality of these has been debated. A few examples and examination of the references therein will lead the reader to many others. Examples are [[Retinex]], an [[artificial neural network]]<ref name="Funt1996">Brian Funt, Vlad Cardei, and Kobus Barnard, "[http://www.cs.sfu.ca/~colour/publications/ARIZONA/arizona_abs.html Learning color constancy]", in ''Proceedings of the Fourth IS&T/SID Color Imaging Conference'', pp. 58–60 (1996).</ref> or a [[Bayesian method]].<ref name=Finlayson2001>{{Cite journal
 |author1=Graham Finlayson |author2=Paul M. Hubel |author3=Steven Hordley |date=November 2001
 | title = Color by correlation: a simple, unifying framework for color constancy
 | journal = [[IEEE Transactions on Pattern Analysis and Machine Intelligence]]
 | volume = 23
 | issue = 11
 | pages = 1209–21
 | doi = 10.1109/34.969113
 | url = http://www2.cmp.uea.ac.uk/Research/compvis/Papers/FinHorHub_PAMI01.pdf
|citeseerx=10.1.1.133.2101 }}</ref>

==Chromatic colors==

Color balancing an image affects not only the neutrals, but other colors as well. An image that is not color balanced is said to have a [[color cast]], as everything in the image appears to have been shifted towards one color.<ref name="Yule1967">John A C Yule, ''Principles of Color Reproduction.'' New York: Wiley, 1967.</ref>{{Page needed|date=September 2010}} Color balancing may be thought in terms of removing this color cast.

Color balance is also related to [[color constancy]]. Algorithms and techniques used to attain color constancy are frequently used for color balancing, as well. Color constancy is, in turn, related to [[chromatic adaptation]]. Conceptually, color balancing consists of two steps: first, determining the [[standard illuminant|illuminant]] under which an image was captured; and second, scaling the components (e.g., R, G, and B) of the image or otherwise transforming the components so they conform to the viewing illuminant.

Viggiano found that white balancing in the camera's native [[RGB color model]] tended to produce less color inconstancy (i.e., less distortion of the colors) than in monitor RGB for over 4000 hypothetical sets of camera sensitivities.<ref name="Viggiano2004"/> This difference typically amounted to a factor of more than two in favor of camera RGB. This means that it is advantageous to get color balance right at the time an image is captured, rather than edit later on a monitor. If one must color balance later, balancing the [[Raw image format|raw image data]] will tend to produce less distortion of chromatic colors than balancing in monitor RGB.

==Mathematics of color balance==
Color balancing is sometimes performed on a three-component image (e.g., [[RGB color model|RGB]]) using a 3x3 [[matrix (mathematics)|matrix]]. This type of transformation is appropriate if the image was captured using the wrong white balance setting on a digital camera, or through a color filter. Changing the color balance of an image can improve classifier results on a trained ML model.

===Scaling monitor R, G, and B===
In principle, one wants to scale all relative luminances in an image so that objects which are believed to be [[grey|neutral]] appear so. If, say, a surface with <math>R=240</math> was believed to be a white object, and if 255 is the count which corresponds to white, one could multiply all [[red]] values by 255/240. Doing analogously for [[green]] and [[blue]] would result, at least in theory, in a color balanced image. In this type of transformation the 3x3 matrix is a [[diagonal matrix]].

: <math>\left[\begin{array}{c} R \\ G \\ B \end{array}\right]=\left[\begin{array}{ccc}255/R'_w & 0 & 0 \\ 0 & 255/G'_w & 0 \\ 0 & 0 & 255/B'_w\end{array}\right]\left[\begin{array}{c}R' \\ G' \\ B' \end{array}\right]</math>

where <math>R</math>, <math>G</math>, and <math>B</math> are the color balanced red, green, and blue components of a [[pixel]] in the image; <math>R'</math>, <math>G'</math>, and <math>B'</math> are the red, green, and blue components of the image before color balancing, and <math>R'_w</math>, <math>G'_w</math>, and <math>B'_w</math> are the red, green, and blue components of a pixel which is believed to be a white surface in the image before color balancing. This is a simple scaling of the red, green, and blue channels, and is why color balance tools in [[Photoshop]] have a white eyedropper tool.  It has been demonstrated that performing the white balancing in the phosphor set assumed by [[sRGB]] tends to produce large errors in chromatic colors, even though it can render the neutral surfaces perfectly neutral.<ref name="Viggiano2004">{{cite book|doi=10.1117/12.524922|chapter=Comparison of the accuracy of different white-balancing options as quantified by their color constancy|title=Sensors and Camera Systems for Scientific, Industrial, and Digital Photography Applications V|volume=5301|pages=323–333|year=2004|last1=Viggiano|first1=J A Stephen|s2cid=8971750|editor3-first=Ricardo J|editor3-last=Motta|editor2-first=Nitin|editor2-last=Sampat|editor1-first=Morley M|editor1-last=Blouke}}</ref>

===Scaling X, Y, Z===
If the image may be transformed into [[CIE 1931 color space|CIE XYZ tristimulus values]], the color balancing may be performed there.  This has been termed a "wrong von Kries" transformation.<ref name=Terstiege1972>{{Cite journal
 | author = Heinz Terstiege
 | title = Chromatic adaptation: a state-of-the-art report
 | year = 1972
 | journal = Journal of Color Appearance
 | volume = 1
 | issue = 4
 | pages = 19–23 (cont. 40)
}}</ref><ref name="Fairchild1998">Mark D Fairchild, ''Color Appearance Models.'' Reading, MA: Addison-Wesley, 1998.</ref> Although it has been demonstrated to offer usually poorer results than balancing in monitor RGB, it is mentioned here as a bridge to other things. Mathematically, one computes:

:<math>\left[\begin{array}{c} X \\ Y \\ Z \end{array}\right]=\left[\begin{array}{ccc}X_w/X'_w & 0 & 0 \\ 0 & Y_w/Y'_w & 0 \\ 0 & 0 & Z_w/Z'_w\end{array}\right]\left[\begin{array}{c}X' \\ Y' \\ Z' \end{array}\right]</math>

where <math>X</math>, <math>Y</math>, and <math>Z</math> are the color-balanced tristimulus values; <math>X_w</math>, <math>Y_w</math>, and <math>Z_w</math> are the tristimulus values of the viewing illuminant (the white point to which the image is being transformed to conform to); <math>X'_w</math>, <math>Y'_w</math>, and <math>Z'_w</math> are the tristimulus values of an object believed to be white in the un-color-balanced image, and <math>X'</math>, <math>Y'</math>, and <math>Z'</math> are the tristimulus values of a pixel in the un-color-balanced image. If the tristimulus values of the monitor primaries are in a matrix <math>\mathbf{P}</math> so that:

:<math>\left[\begin{array}{c} X \\ Y \\ Z \end{array}\right]=\mathbf{P}\left[\begin{array}{c}L_R \\ L_G \\ L_B \end{array}\right]</math>

where <math>L_R</math>, <math>L_G</math>, and <math>L_B</math> are the un-[[gamma correction|gamma corrected]] monitor RGB, one may use:

:<math>\left[\begin{array}{c} L_R \\ L_G \\ L_B \end{array}\right]=\mathbf{P^{-1}}\left[\begin{array}{ccc}X_w/X'_w & 0 & 0 \\ 0 & Y_w/Y'_w & 0 \\ 0 & 0 & Z_w/Z'_w\end{array}\right]\mathbf{P}\left[\begin{array}{c}L_{R'} \\ L_{G'} \\ L_{B'} \end{array}\right]</math>

===Von Kries's method===
[[Johannes von Kries]], whose theory of [[rod cell|rods]] and three color-sensitive [[cone cell|cone]] types in the [[retina]] has survived as the dominant explanation of color sensation for over 100 years, motivated the method of converting color to the [[LMS color space]], representing the effective stimuli for the Long-, Medium-, and Short-wavelength cone types that are modeled as adapting independently.  A 3x3 matrix converts RGB or XYZ to LMS, and then the three LMS primary values are scaled to balance the neutral; the color can then be converted back to the desired final [[color space]]:<ref name=Sharma>{{Cite book| title = Digital Color Imaging Handbook | author =  Gaurav Sharma| url = https://books.google.com/books?id=AkByHKRGTsQC&q=%22von+Kries%22&pg=PA153 | publisher = [[CRC Press]] | year = 2003 | isbn = 978-0-8493-0900-7 | page=153 }}</ref>

:<math>\left[\begin{array}{c} L \\ M \\ S \end{array}\right]=\left[\begin{array}{ccc}1/L'_w & 0 & 0 \\ 0 & 1/M'_w & 0 \\ 0 & 0 & 1/S'_w\end{array}\right]\left[\begin{array}{c}L' \\ M' \\ S' \end{array}\right]</math>
where <math>L</math>, <math>M</math>, and <math>S</math> are the color-balanced LMS cone tristimulus values; <math>L'_w</math>, <math>M'_w</math>, and <math>S'_w</math> are the tristimulus values of an object believed to be white in the un-color-balanced image, and <math>L'</math>, <math>M'</math>, and <math>S'</math> are the tristimulus values of a pixel in the un-color-balanced image.

Matrices to convert to LMS space were not specified by von Kries, but can be derived from CIE color matching functions and LMS color matching functions when the latter are specified; matrices can also be found in reference books.<ref name=Sharma/>

===Scaling camera RGB===
By Viggiano's measure, and using his model of gaussian camera spectral sensitivities, most camera RGB spaces performed better than either monitor RGB or XYZ.<ref name="Viggiano2004"/>  If the camera's raw RGB values are known, one may use the 3x3 diagonal matrix:

: <math>\left[\begin{array}{c} R \\ G \\ B \end{array}\right]=\left[\begin{array}{ccc}255/R'_w & 0 & 0 \\ 0 & 255/G'_w & 0 \\ 0 & 0 & 255/B'_w\end{array}\right]\left[\begin{array}{c}R' \\ G' \\ B' \end{array}\right]</math>

and then convert to a working RGB space such as [[sRGB]] or [[Adobe RGB]] after balancing.
<!-- in progress!
However, if one has already converted to monitor RGB, one may still work in camera RGB if a 3x3 [[regular matrix]] <math>\mathbf{A}</math> characterizes the camera's color mixing behavior reasonably well, so that:

: <math>\left[\begin{array}{c} X \\ Y \\ Z \end{array}\right]\approx\mathbf{A}\left[\begin{array}{c}L_R \\ L_G \\ L_B \end{array}\right]</math>

(This matrix is included in the [[ICC profile]] for some cameras.<ref name="ICC01_2006">International Color Consortium, ''Specification ICC.1:2004-10 (Profile version 4.2.0.0) Image technology colour management — Architecture, profile format, and data structure'', (2006).</ref>) If this matrix is known, one computes:

:<math>\left[\begin{array}{c} L_R \\ L_G \\ L_B \end{array}\right]=\mathbf{P^{-1}A^{-1}}\left[\begin{array}{ccc}R_w/R'_w & 0 & 0 \\ 0 & G_w/G'_w & 0 \\ 0 & 0 & B_w/B'_w\end{array}\right]\mathbf{A\cdotP}\left[\begin{array}{c}L_{R'} \\ L_{G'} \\ L_{B'} \end{array}\right]</math>

where <math>\mathbf{P}</math> is the phosphor matrix mentioned in the previous section; <math>L_{Rw}</math, <math>G_w</math, <math>B_w</math are the
-->

===Preferred chromatic adaptation spaces===
Comparisons of images balanced by diagonal transforms in a number of different RGB spaces have identified several such spaces that work better than others, and better than camera or monitor spaces, for chromatic adaptation, as measured by several [[color appearance model]]s; the systems that performed statistically as well as the best on the majority of the image test sets used were the "Sharp", "Bradford", "CMCCAT", and "ROMM" spaces.<ref>{{Cite journal |url=http://infoscience.epfl.ch/getfile.py?recid=34049&mode=best |title=Chromatic Adaptation Performance of Different RGB Sensors |author1=Sabine Süsstrunk |author2=Jack Holm |author3=Graham D. Finlayson |editor-first1=Reiner |editor-first2=Gabriel G. |editor-last1=Eschbach |editor-last2=Marcu |journal=IS&T/SPIE Electronic Imaging |series=Color Imaging: Device-Independent Color, Color Hardcopy, and Graphic Arts VI |volume=4300 |date=January 2001 |pages=172–183 |access-date=2009-03-20 |archive-url=https://web.archive.org/web/20061018020916/http://infoscience.epfl.ch/getfile.py?mode=best&recid=34049 |archive-date=2006-10-18 |url-status=dead |doi=10.1117/12.410788 |s2cid=8140548 }}</ref>

===General illuminant adaptation===
The best color matrix for adapting to a change in illuminant is not necessarily a diagonal matrix in a fixed color space.  It has long been known that if the space of illuminants can be described as a linear model with ''N'' basis terms, the proper color transformation will be the weighted sum of ''N'' fixed linear transformations, not necessarily consistently diagonalizable.<ref>{{Cite book | author1 = Laurence T. Maloney | author2 = Brain A. Wandell | chapter = Color constancy: a method for recovering surface spectral reflectance | title = Readings in Computer Vision | editor1 = Martin A. Fischler | editor2 = Oscar Firschein | year = 1987 | publisher = Morgan-Kaufmann | isbn = 978-0-934613-33-0 | chapter-url = https://books.google.com/books?id=W5hLHUI8U-kC&q=maloney+wandell&pg=PA293 | url = https://archive.org/details/readingsincomput00fisc }}</ref>

=== Examples ===
{{Multiple image
| align = center
| direction = 
| width = 300
| footer = Comparison of resulted colors as shot by the digital camera for different light qualities (color temperature): Neutral, Warm and Cold.<ref name="photoskop">{{cite web |url=http://www.photoskop.com/player.html?l=wb&ch=0&sec=0 |title=photoskop: Interactive Photography Lessons |date=April 25, 2015}}</ref>
| image1 = Wb girl neutral.jpg
| caption1 = Neutral light
| image2 = Wb girl warm.jpg
| caption2 = Warm light
| image3 = Wb girl cold.jpg
| caption3 = Cold light
}}{{Multiple image
| align = center
| direction = 
| width = 300
| footer = Example of different white balance settings on digital camera for Neutral light.<ref name="photoskop"/>
| image1 = Wb girl neutral.jpg
| caption1 = Setting: As shot
| image2 = Wb girl cloudy.jpg
| caption2 = Setting: Cloudy
| image3 = Wb girl tungsten.jpg
| caption3 = Setting: Tungsten
}}

==See also==
* [[Color cast]]
* [[Color temperature]]
* [[Gamma correction]]
* [[White point]]

==References==
{{Reflist|35em}}

==External links==
* [http://www.nikondigital.org/articles/white_balance.htm White Balance] - Intro at nikondigital.org
* [http://www.photoskop.com/player.html?l=wb&ch=0&sec=0 photoskop: Interactive Photography Lessons] - Interactive White Balance
* [https://web.archive.org/web/20080203165737/http://www.photoxels.com/tutorial_white-balance.html Understanding White Balance] - Tutorial
* [http://www.ipol.im/pub/algo/lmps_simplest_color_balance/ Affine color balance with saturation, with code and on-line demonstration]
* [http://www.geofflawrence.com/white_balance.html Getting the White Balance Right for Neutral Colors] - Photography Tutorial

{{Color topics}}

[[Category:Color|balance]]
[[Category:Image processing]]
[[Category:Film post-production]]