conclusion.tex

% ---------------------------------------------------------------------------
\chapter{Conclusions and Future Research}\label{ch:conclusion}
% ---------------------------------------------------------------------------
This research has demonstrated that a combination of better component specifications achieved with a flexible and expressive mathematical system and a minimalist prover for dispatching the sorts of VCs that arise from well-engineered programs can significantly impact the verifiability of software.  While faster and more advanced provers have been persued to address the software verification problem, this research has demonstrated that an extremely modest prover can compete with more traditional verification systems simply by exploiting properties and patterns inherent to software verification.  In short, we have confirmed that programmers write code they believe works, and that, with suitable formal documentation, the necessary insights for proving resulting proof obligations should therefore be shallow.

Beyond showing that automatic formal reasoning about programs is possible, this research has led to the development of a prototype verification system that exemplifies a hybrid design philosophy.  It takes many of the best attributes of purely mathematical reasoning systems and blends them with a practical, imperative, object-based programming language.  As a result, RESOLVE now represents the only verification language to combine an imperative, object-based language with features like higher-order logic, first-class dependent types, and an extensible mathematical universe based on a traditional foundational theory.  In the process, we have designed and implemented a novel compromise for type-checking a language with dependent types.  We have experimented with the type system and the prover in classroom education and validated them with a range of research benchmarks.

While this research has taken an important step toward addressing the verification problem, we acknowledge that it represents only a small step toward the eventual goal of realizing a true push-button verifying compiler.  Several questions remain:

While our ideas on how to qualify a likely ``good'' step were effective in bringing additional VCs into the realm of provability, for those VCs that were provable without such a prioritization heuristic, it caused a significant degradation of performance.  The consistency with which it did so nonetheless suggests that a better function is out there, as we have one that describes ``inefficient'' steps.  An open question is to determine if our function fails to take into account important variables, or if it merely combines the variables it has in the wrong proportions.  One interesting experiment might be to use a genetic algorithm or some other general optimization technique to attempt to find a suitable function.

In Section~\ref{sec:reversingAQueue} we present a brief exploration of how alternative styles of specification affect the verifiability of a component.  While we did not find any particular advantage to implicit or explicit style of specification, many other such dichotomies exist.  For example, we have discussed elsewhere how the choice of mathematical model may have a large impact on the verifiability of a component in \cite{smithSpecificationAbstractions}.  Larger-scale experiments on these dimensions of specification variation may lead to interesting new insights.

Similarly, while in this work we have developed a minimalist prover in term-rewrite style, other straightforward sorts of provers exist.  An example of this is a Gentzen-style prover such as in \cite{manna1995step}, which strives to build up a set of knowns and a set of assertions sufficient to complete the proof until these sets overlap.  With large-scale experimentation, advantages may be discovered among different styles of provers that inform an appropriate prover design for the verification task.  The architecture of the prover presented in this work has been designed with such prover flexibility in mind and our current backtracking-model could be replaced with, e.g., a Gentzen-style prover without much difficulty.

Consistent with our work at Clemson applying formal reasoning as an educational tool, this research has been undertaken with educational applications in mind.  While some of our tools have been used extensively in classroom settings, an important area of future work is to evaluate the system as a whole and many of the newer features in such an environment so that the benefits on student learning can be analyzed and validated.

While we believe that our static typing system is now the most flexible of any language, practical or pure, research question remain regarding that flexibility.  Despite a straightforward algorithm, mixing the use of explicit type parameters, implicit type parameters, and type theorems occasionally produces counter-intuitive results, causing statements that are certainly mathematically sound to fail type-checking.  A subtler algorithm based on a dependency tree rather than strict left-to-right evaluation might be able to statically establish such statements.  Similarly, when mixing multiple types with complex relationships, RESOLVE is often unable to break ties between multiple seemingly-equally-applicable functions.  A mechanism to prioritize those by giving the type-checking system insight into the relationships between \emph{functions} in addition to types would resolve many of these issues.

The scalability of the prover is always a serious concern.  After all, both the antecedent development step and the consequent exploration step suffer from combinatorial explosion---the former of space, the latter of time.  While we are heartened to find that, after the application of our heuristics, very few steps are generally required in the consequent exploration phase, this does little to address the issue with the antecedent development stage.  To maintain scalability, further research will need to be done to qualify useful developments and useless transformations.  We hypothosize that ultimately a feedback loop between consequent exploration and antecedent development might be more appropriate than our current two-phase algorithm.  This would represent a kind of simulated annealing wherein the consequent exploration would represent a random walk, and intermittent antecedent development would function as a hill-climb.

Finally, and most importantly, more research is required on the human component of verification.  We are excited to have contributed to this with a more intuitive mathematical system with which---we hope---a mathematician will feel more at home than with the currently available automation-friendly mathematical languages.  But the interaction of a human programmer with a programming system is a complex and under-appreciated problem in which we often do not see the programmers of the language as true users.  But if verification is to succeed, we must overcome these issues to develop a system that strikes the proper balance between the formal rigor required for successful verification and the insights needed to support programmers and bring that rigor within reach.