Editing Symbolic execution (section)

=== Environment interactions ===
Programs interact with their environment by performing [[system call]]s, receiving signals, etc. Consistency problems may arise when execution reaches components that are not under control of the symbolic execution tool (e.g., kernel or libraries). Consider the following example:<syntaxhighlight lang="c" line="1">
int main()
{
  FILE *fp = fopen("doc.txt");
  ...
  if (condition) {
    fputs("some data", fp);
  } else {
    fputs("some other data", fp);
  }
  ...
  data = fgets(..., fp);
}
</syntaxhighlight>This program opens a file and, based on some condition, writes different kind of data to the file. It then later reads back the written data. In theory, symbolic execution would fork two paths at line 5 and each path from there on would have its own copy of the file. The statement at line 11 would therefore return data that is consistent with the value of "condition" at line 5. In practice, file operations are implemented as system calls in the kernel, and are outside the control of the symbolic execution tool. The main approaches to address this challenge are:

'''Executing calls to the environment directly.''' The advantage of this approach is that it is simple to implement. The disadvantage is that the side effects of such calls will clobber all states managed by the symbolic execution engine. In the example above, the instruction at line 11 would return "some datasome other data" or "some other datasome data" depending on the sequential ordering of the states.

'''Modeling the environment.''' In this case, the engine instruments the system calls with a model that simulates their effects and that keeps all the side effects in per-state storage. The advantage is that one would get correct results when symbolically executing programs that interact with the environment. The disadvantage is that one needs to implement and maintain many potentially complex models of system calls. Tools such as KLEE,<ref>{{Cite journal|title = KLEE: Unassisted and Automatic Generation of High-coverage Tests for Complex Systems Programs|url = http://dl.acm.org/citation.cfm?id=1855741.1855756|journal = Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation|date = 2008-01-01|pages = 209–224|series = OSDI'08|first1 = Cristian|last1 = Cadar|first2 = Daniel|last2 = Dunbar|first3 = Dawson|last3 = Engler}}</ref> Cloud9, and Otter<ref>{{cite web|title=MultiOtter: Multiprocess Symbolic Execution|url=https://www.cs.umd.edu/~mwh/papers/multiotter.pdf|first1 = Jonathan|last1 = Turpie|first2 = Elnatan|last2 = Reisner|first3 = Jeffrey|last3 = Foster|first4 = Michael |last4 = Hicks}}</ref> take this approach by implementing models for file system operations, sockets, [[Inter-process communication|IPC]], etc.

'''Forking the entire system state.''' Symbolic execution tools based on virtual machines solve the environment problem by forking the entire VM state. For example, in S2E<ref>{{Cite journal|title = The S2E Platform: Design, Implementation, and Applications|journal = ACM Trans. Comput. Syst.|date = 2012-02-01|issn = 0734-2071|pages = 2:1–2:49|volume = 30|issue = 1|doi = 10.1145/2110356.2110358|first1 = Vitaly|last1 = Chipounov|first2 = Volodymyr|last2 = Kuznetsov|first3 = George|last3 = Candea|s2cid = 16905399| url=http://infoscience.epfl.ch/record/175469 }}</ref> each state is an independent VM snapshot that can be executed separately. This approach alleviates the need for writing and maintaining complex models and allows virtually any program binary to be executed symbolically. However, it has higher memory usage overheads (VM snapshots may be large).