Evaluating Contextualized Code Search in Practical User Studies
| dc.contributor.author | Villmow, Johannes | |
| dc.contributor.author | Ulges, Adrian | |
| dc.contributor.author | Schwanecke, Ulrich | |
| dc.contributor.editor | Klein, Maike | |
| dc.contributor.editor | Krupka, Daniel | |
| dc.contributor.editor | Winter, Cornelia | |
| dc.contributor.editor | Gergeleit, Martin | |
| dc.contributor.editor | Martin, Ludger | |
| dc.date.accessioned | 2024-10-21T18:24:13Z | |
| dc.date.available | 2024-10-21T18:24:13Z | |
| dc.date.issued | 2024 | |
| dc.description.abstract | Contextualized Code Search (CCS) aims to retrieve relevant code snippets that complement the developer’s current editor context. In contrast to AI-based code generation, it offers the key benefit that the source of the retrieved code is made transparent, allowing for a safe re-use of code within companies. Recently, self-supervised training for CCS has been shown to be effective. Evidence for this, however, focuses on ranking quality on research datasets. It remains unclear whether – and if yes, by how far – CCS can help improve the efficiency of real-world users. To fill this gap, we have integrated a recent CCS model into an IDE. We describe specialized robustness-oriented enhancements to the training to improve usability. We then evaluate the model in two practical user studies: In Study A, we measure efficiency improvements of fourth semester computer science students on simple algorithm exercises. In Study B, we allow a professional software development team to use the tool in their everyday work. Their company consists of several – more or less independent – teams that work on the same product, which might find code of other teams helpful. We demonstrate improvements by the proposed search, discuss use cases for the tool, and point out challenges and directions for future research (such as the combination with code generation in retrieval augmented generation). | en |
| dc.identifier.doi | 10.18420/inf2024_122 | |
| dc.identifier.eissn | 2944-7682 | |
| dc.identifier.isbn | 978-3-88579-746-3 | |
| dc.identifier.issn | 2944-7682 | |
| dc.identifier.pissn | 1617-5468 | |
| dc.identifier.uri | https://dl.gi.de/handle/20.500.12116/45095 | |
| dc.language.iso | en | |
| dc.publisher | Gesellschaft für Informatik e.V. | |
| dc.relation.ispartof | INFORMATIK 2024 | |
| dc.relation.ispartofseries | Lecture Notes in Informatics (LNI) - Proceedings, Volume P-352 | |
| dc.subject | Contextualized Code Search | |
| dc.subject | Code Retrieval | |
| dc.subject | User Studies | |
| dc.subject | Self-supervised Learning | |
| dc.title | Evaluating Contextualized Code Search in Practical User Studies | en |
| dc.type | Text/Conference Paper | |
| gi.citation.endPage | 1405 | |
| gi.citation.publisherPlace | Bonn | |
| gi.citation.startPage | 1395 | |
| gi.conference.date | 24.-26. September 2024 | |
| gi.conference.location | Wiesbaden | |
| gi.conference.sessiontitle | AI@WORK |
Dateien
Originalbündel
1 - 1 von 1
Lade...
- Name:
- Villmow_et_al_Evaluating_Contextualized_Code_Search.pdf
- Größe:
- 1.63 MB
- Format:
- Adobe Portable Document Format