Konferenzbeitrag
Evaluating Contextualized Code Search in Practical User Studies
Lade...
Volltext URI
Dokumententyp
Text/Conference Paper
Zusatzinformation
Datum
2024
Autor:innen
Zeitschriftentitel
ISSN der Zeitschrift
Bandtitel
Quelle
Verlag
Gesellschaft für Informatik e.V.
Zusammenfassung
Contextualized Code Search (CCS) aims to retrieve relevant code snippets that complement the developer’s current editor context. In contrast to AI-based code generation, it offers the key benefit that the source of the retrieved code is made transparent, allowing for a safe re-use of code within companies. Recently, self-supervised training for CCS has been shown to be effective. Evidence for this, however, focuses on ranking quality on research datasets. It remains unclear whether – and if yes, by how far – CCS can help improve the efficiency of real-world users. To fill this gap, we have integrated a recent CCS model into an IDE. We describe specialized robustness-oriented enhancements to the training to improve usability. We then evaluate the model in two practical user studies: In Study A, we measure efficiency improvements of fourth semester computer science students on simple algorithm exercises. In Study B, we allow a professional software development team to use the tool in their everyday work. Their company consists of several – more or less independent – teams that work on the same product, which might find code of other teams helpful. We demonstrate improvements by the proposed search, discuss use cases for the tool, and point out challenges and directions for future research (such as the combination with code generation in retrieval augmented generation).