Editing Generative model (section)

== Deep generative models ==
With the rise of [[deep learning]], a new family of methods, called deep generative models (DGMs),<ref name="auto1">{{Cite web|url=https://www.microsoft.com/en-us/research/blog/a-deep-generative-model-trifecta-three-advances-that-work-towards-harnessing-large-scale-power/|title=Scaling up—researchers advance large-scale deep generative models|website=[[Microsoft]] |date=April 9, 2020}}</ref><ref name="auto">{{Cite web|url=https://openai.com/blog/generative-models/|title=Generative Models|date=June 16, 2016|website=OpenAI}}</ref> is formed through the combination of generative models and deep neural networks. An increase in the scale of the neural networks is typically accompanied by an increase in the scale of the training data, both of which are required for good performance.<ref>{{Cite arXiv |eprint = 2001.08361|last1 = Kaplan|first1 = Jared|last2 = McCandlish|first2 = Sam|last3 = Henighan|first3 = Tom|last4 = Brown|first4 = Tom B.|last5 = Chess|first5 = Benjamin|last6 = Child|first6 = Rewon|last7 = Gray|first7 = Scott|last8 = Radford|first8 = Alec|last9 = Wu|first9 = Jeffrey|last10 = Amodei|first10 = Dario|title = Scaling Laws for Neural Language Models|year = 2020|class = stat.ML}}</ref>

Popular DGMs include [[Autoencoder#Variational autoencoder (VAE)|variational autoencoders]] (VAEs), [[generative adversarial networks]] (GANs), and auto-regressive models. Recently, there has been a trend to build very large deep generative models.<ref name="auto1"/> For example, [[GPT-3]], and its precursor [[GPT-2]],<ref>{{Cite web|url=https://openai.com/blog/better-language-models/|title=Better Language Models and Their Implications|date=February 14, 2019|website=OpenAI}}</ref> are auto-regressive neural language models that contain billions of parameters, BigGAN<ref>{{Cite arXiv |eprint = 1809.11096|last1 = Brock|first1 = Andrew|last2 = Donahue|first2 = Jeff|last3 = Simonyan|first3 = Karen|title = Large Scale GAN Training for High Fidelity Natural Image Synthesis|year = 2018|class = cs.LG}}</ref> and VQ-VAE<ref>{{Cite arXiv |eprint = 1906.00446|last1 = Razavi|first1 = Ali|last2 = van den Oord|first2 = Aaron|last3 = Vinyals|first3 = Oriol|title = Generating Diverse High-Fidelity Images with VQ-VAE-2|year = 2019|class = cs.LG}}</ref> which are used for image generation that can have hundreds of millions of parameters, and Jukebox is a very large generative model for musical audio that contains billions of parameters.<ref>{{Cite web|url=https://openai.com/blog/jukebox/|title=Jukebox|date=April 30, 2020|website=OpenAI}}</ref>