Program
Elizabeth Polgreen University of Edinburgh, UK
Keynote: Making the most of large language models for program synthesis (when you want correct answers)Abstract. Since the release of ChatGPT, large language models (LLMs) have been dominating the discourse around code generation. In contrast, the best-performing solvers in the world of formal program synthesis are still based around enumerative algorithms. So, are we behind the times? Are LLMs the answer to all our synthesis problems? In this talk, I will present two approaches to making the most of LLMs in formal domains. First, I will present a technique based on combining enumerative program synthesis with guidance from an LLM [1]. Focusing on the Syntax-Guided Synthesis (SyGuS) competition benchmarks, we found that, whilst LLMs can produce syntactically correct SyGuS expressions, they fail to produce semantically correct solutions for more than half of the benchmarks. We propose a novel enumerative synthesis algorithm, which integrates calls to an LLM into a weighted probabilistic search, allowing 2-way exchange of information between the two components, and increasing the number of benchmarks solved to 80% (and outperforming the state-of-the-art solver). Second, I will discuss our work using LLMs to generate models of systems for verification [2]. Here, we enable LLMs to generate syntactically correct code, even in programming languages that barely feature in the training data, by combining an HCI technique called natural program elicitation and a Max-SMT solver. We apply our approach to generating models of systems in the UCLID5 verification language, improving syntactic correctness from <10% to >80% on a set of textbook problems. Whilst I don’t believe LLMs are the answer to all our synthesis problems, I think our results demonstrate that perhaps techniques used in formal synthesis could be the answer to many of the problems facing LLMs..!
[1] Guiding Enumerative Program Synthesis with Large Language Models Yixuan Li, Julian Parsert, Elizabeth Polgreen
[2] Synthetic Programming Elicitation and Repair for Text-to-Code in Very Low-Resource Programming Languages - Federico Mora, Justin Wong, Haley Lepe, Sahil Bhatia, Karim Elmaaroufi, George Varghese, Joseph E. Gonzalez, Elizabeth Polgreen, Sanjit A. Seshia
Bio. Dr Elizabeth Polgreen is a lecturer (~Assistant Professor in US terminology) in the School of Informatics at the University of Edinburgh. Prior to this, she was a postdoctoral research scholar in Professor Sanjit Seshia’s group at the University of California, Berkeley, and completed a PhD at the University of Oxford, supervised by Professor Alessandro Abate. She is interested in formal program synthesis techniques, combinations of formal program synthesis and machine learning driven synthesis, and the use of synthesis to increase the scalability of verification. She currently holds a research fellowship from the Royal Academy of Engineering.
Ashutosh Trivedi University of Colorado, Boulder
Keynote: Expanding Horizons: Hyperproperties in CPS, Fairness, and Legal Compliance RequirementsAbstract. In recent years, the study of hyperproperties has provided profound insights into complex system behaviors, yet many practical fields remain unexplored territories for these concepts. This talk aims to highlight the connection between hyperproperty research and real-world applications, drawing from my work in related domains such as cyber-physical systems (CPS), software fairness, legal compliance, and metamorphic testing. I will illustrate how concepts developed for hyperproperties can be effectively applied to ensure system confidentiality, equitable outcomes, regulatory adherence, and robust software testing. By exploring these practical settings, I hope to invite formal methods researchers to venture into new areas, offering fresh challenges and groundbreaking possibilities for the application of their innovative research. Join me as we discuss these diverse fields, uncovering the potential for hyperproperties to revolutionize approaches and solutions across various disciplines.
Bio. Ashutosh Trivedi is an Associate Professor of Computer Science at the University of Colorado Boulder, specializing in the development and application of formal methods for the design and analysis of safety-critical learning-enabled systems. He obtained his doctoral degree in computer science from the University of Warwick, with a specialization in game theory and optimization. Prior to his current position, Ashutosh served as an Assistant Professor of Computer Science at the Indian Institute of Technology Bombay. He also worked as a Postdoctoral Research Associate at the University of Pennsylvania and the University of Oxford. Ashutosh is a recipient of the 2022 NSF CAREER award and a Liverpool Fellowship. His research interests lie at the intersection of computer science, control theory, and machine learning, with a focus on formal methods and reinforcement learning.
09:00-10:00 |
Keynote: Making the most of large language models for program synthesis (when you want correct answers) Elizabeth Polgreen (University of Edinburgh) |
10:00-10:15 |
Coffee Break |
10:10-10:45 |
Can LLMs Perform Verified Lifting of Code? Sahil Bhatia, Jie Qiu, Niranjan Hasabnis, Sanjit Seshia and Alvin Cheung Domain-specific languages (DSLs) have become integral to various software workflows. Such languages offer domain-specific optimizations and abstractions that improve code readability and maintainability. However, leveraging these languages requires developers to rewrite existing code using the specific DSL's API. While large language models (LLMs) have shown some success in automatic code transpilation, none of them provide any functional correctness guarantees on the rewritten code. Another approach for automating this task is verified lifting, which relies on program synthesis to find programs in the target language that are functionally equivalent to the source language program. While several verified lifting tools have been developed for various application domains, they are specialized for specific source-target languages or require significant expertise in domain knowledge to make the search efficient. In this paper, leveraging recent advances in LLMs, we propose an LLM-based approach to building verified lifting tools. We use the LLM's capabilities to reason about programs to translate a given program into its corresponding equivalent in the target language. Additionally, we use LLMs to generate proofs for functional equivalence. We develop lifting-based compilers for three DSLs targeting different application domains. Our approach outperforms previous symbolic-based tools in the number of benchmarks transpiled and also requires significantly less efforts to build. |
10:45-11:15 |
Reactive Synthesis Modulo Theories with Data Temporal Relations (Extended Abstract) Andoni Rodriguez, César Sánchez and Felipe Gorostiaga Recently, the realizability problem of LTL Modulo Theories ($\LTLt$) specifications has been shown to be decidable whenever theories are decidable. However, such approaches suffer from a key limitation: memory of specifications is restricted. They allow to carry finite information of infinite theories. For instance, we can remember that "x" was odd in previous timesteps, but we cannot remember the concrete value of "x" in previous timesteps. This limits the ability of abstractions to properly encode richer dynamics in the specifications (e.g., it is currently not possible to specify that a variable increases monotonically over time). In this paper, we extend $\LTLt$ with operators that allow to fetch values of the variables in previous instants ($\LTLtf$). We show that realizability in $\LTLtf$ is undecidable in general, but we also propose a sound incomplete algorithm that works for expressive $\LTLtf$ formulae in which we can specify realistic environment dynamics. In addition, we present a novel procedure that abstracts tautologies in $\LTLtf$ to tautologies in $\LTLt$, which conveys memory across-time and thus helps pushing conclusiveness of our algorithm. Moreover, we show that we can encode inductive behaviour such as counters and averages. To the best of our knowledge, this is the first work that presents sound realizability algorithms that conclude in such expressive specifications. |
11:15-11:45 |
Synthesizing Exact Loop Bounds Daniel Riley and Grigory Fedyukovich Many state-of-the-art techniques are designed to automatically infer upper limits on the number of iterations a loop may execute, conventionally called bounds. Analysis of this kind is inspired by techniques to prove termination used in previous works, namely to instrument the program with a decrementing counter and find the conditions by which the counter’s value is above zero at the end of the program. However, in most cases, they do not capture the exact number of iterations which could be useful to estimate the number of cycles/iterations/time a process requires when it is distributed across many devices, relational verification and equivalence checking, and even non-termination. In this extended abstract, we present the ongoing work on synthesis of an exact loop bound for single-loop programs as a function over the program’s inputs. |
11:45-14:00 |
Lunch |
14:00-15:00 |
Keynote: Expanding Horizons: Hyperproperties in CPS, Fairness, and Legal Compliance Requirements Ashutosh Trivedi (University of Colorado, Boulder) |
15:00-15:15 |
Coffee Break |
15:15-15:45 |
ZDD Boolean Synthesis Yi Lin, Lucas M. Tabajara, Moshe Y. Vardi Motivated by applications in boolean-circuit design, boolean synthesis is the process of synthesizing a boolean function with multiple outputs, given a relation between its inputs and outputs. Previous work has attempted to solve boolean functional synthesis by converting a specification formula into a Binary Decision Diagram (BDD) and quantifying existentially the output variables. We make use of the fact that the specification is usually given in the form of a Conjunctive Normal Form (CNF) formula, and we can perform resolution on a symbolic representation of a CNF formula in the form of a Zero-suppressed Binary Decision Diagram (ZDD). We adapt the realizability test to the context of CNF and ZDD, and show that the Cross operation defined in earlier work can be used for witness construction. Experiments show that our approach is complementary to BDD-based Boolean synthesis. |
15:45-16:15 |
Synthesis of Multi-Threaded Programs Sidi Mohamed Beillahi, Ahmed Bouajjani, Constantin Enea and Shuvendu Lahiri Asynchronous and concurrent programming are fundamental concepts for building responsive and efficient software systems. To this end, we proposed an approach for refactoring a sequential program into an asynchronous program that uses async/await, called asynchronization, without introducing data races [1]. We showed that the delay complexity of enumerating all data race free asynchronizations, which quantifies the delay between outputting two consecutive data race free asynchronizations, is polynomial time modulo an oracle for solving reachability in sequential programs. In this paper, we show that this approach and the associated results extend to refactoring a sequential program into a data race free multi-threaded program that uses start/join. |
16:15-16:35 |
SYNTCOMP results Swen Jacobs This is the presentation of the SYNTCOMP results. |
16:35-16:40 |
Closing |