Editing Data engineering (section)

=== Compute ===
High-performance computing is critical for the processing and analysis of data. One particularly widespread approach to computing for data engineering is [[dataflow programming]], in which the computation is represented as a [[directed graph]] (dataflow graph); nodes are the operations, and edges represent the flow of data.<ref name="sigops">{{cite web |last1=Schwarzkopf |first1=Malte |title=The Remarkable Utility of Dataflow Computing |url=https://www.sigops.org/2020/the-remarkable-utility-of-dataflow-computing/ |website=ACM SIGOPS |access-date=31 July 2022 |date=7 March 2020}}</ref> Popular implementations include [[Apache Spark]], and the [[deep learning]] specific [[TensorFlow]].<ref name="sigops" /><ref name="sparkpaper">{{cite web |url=https://cs.stanford.edu/~matei/papers/2016/cacm_apache_spark.pdf |access-date=31 July 2022|title=sparkpaper}}</ref><ref name="tensorflow paper">{{cite web |last1=Abadi |first1=Martin |last2=Barham |first2=Paul |last3=Chen |first3=Jianmin |last4=Chen |first4=Zhifeng |last5=Davis |first5=Andy |last6=Dean |first6=Jeffrey |last7=Devin |first7=Matthieu |last8=Ghemawat |first8=Sanjay |last9=Irving |first9=Geoffrey |last10=Isard |first10=Michael |last11=Kudlur |first11=Manjunath |last12=Levenberg |first12=Josh |last13=Monga |first13=Rajat |last14=Moore |first14=Sherry |last15=Murray |first15=Derek G. |last16=Steiner |first16=Benoit |last17=Tucker |first17=Paul |last18=Vasudevan |first18=Vijay |last19=Warden |first19=Pete |last20=Wicke |first20=Martin |last21=Yu |first21=Yuan |last22=Zheng |first22=Xiaoqiang |title=TensorFlow: A system for large-scale machine learning |url=https://research.google/pubs/pub45381/ |website=12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16) |access-date=31 July 2022 |pages=265–283 |date=2016}}</ref> More recent implementations, such as [[Differential Dataflow|Differential]]/[[Timely Dataflow|Timely]] Dataflow, have used [[incremental computing]] for much more efficient data processing.<ref name="sigops" /><ref name="differential-paper">{{cite web |last1=McSherry |first1=Frank |last2=Murray |first2=Derek |last3=Isaacs |first3=Rebecca |last4=Isard |first4=Michael |title=Differential dataflow |website=[[Microsoft]] |url=https://www.microsoft.com/en-us/research/publication/differential-dataflow/ |access-date=31 July 2022 |date=5 January 2013}}</ref><ref name="differential-github">{{cite web |title=Differential Dataflow |url=https://github.com/TimelyDataflow/differential-dataflow |publisher=Timely Dataflow |access-date=31 July 2022 |date=30 July 2022}}</ref>