Traditional static analysis tools without learning have existed for decades. So, it begs the question, why should we introduce deep learning? As indicated by Rice's Theorem any non-trivial semantic property of a program is undecidable. Clearly, there are limitations in traditional analysis techniques. When attempting to enforce an undecidable errors will likely arise. These are often in the form of false positives which can be distracting to the user. False positives warn users about code issues that do not occur in practice. False positives are among the major pain points for developers using analysis tools.
Furthermore, traditional analyses often require handwritten rules that target certain error patterns in a particular codebase or specific bug types. They cannot deal with functional bugs (i.e., errors that violate the program specification and yet conform to the coding rules).
The figure below shows one such example. The goal is to split a string using regular expressions. However, the program incorrectly splits the input ' and ' into ['', ' and ', ''] instead of ['' , ''], which is what the developer intended. Since the error is simply a mismatch between the developer’s implicit specification and implementation, static analyzers are incapable of catching it.
So, why is deep learning the solution?
The bugs that static analyzers missed in both cases are in hindsight quite obvious to human programmers. The criteria they use is very simple: any code snippet that seems to deviate from common code patterns is likely to be buggy. This is precisely the observation that our approach seeks to mimic. In particular, if a model observes a property or an unusual way of splitting strings that never appeared in the training data, it is likely to recognize those abnormal code fragments as potential bugs. In this way, learning provides a highly flexible codebase and bug type independent approach to analysis.