Exploring Similarity Assessments of LLM-generated Requirements
R. M. García Alarcia, A. Golkar and A. Salado
23rd Annual Conference on Systems Engineering Research (CSER 2026), 2026
Abstract
A potential application of LLMs in SE is the production of sets of requirements for a particular problem. In this paper, we survey how researchers evaluate the quality of requirements generated by LLMs and assess the adequacy of such approaches. Particularly, we show that metrics traditionally employed to evaluate the similarity of LLM-generated artifacts against ground truths do not perform well for requirements. The underlying reason is that similarity ought not to be achieved on the textual output but on the problem space captured by the requirements.
Keywords: requirements, large language models (LLM), CoPilot, semantic similarity, cosine similarity
Cite
@inproceedings{GarciaAlarcia2026ExploringSimilarity,
author = {Garc{\'\i}a Alarcia, Ram{\'o}n Mar{\'\i}a and Golkar, Alessandro and Salado, Alejandro},
title = {Exploring Similarity Assessments of {LLM-generated} Requirements},
booktitle = {23rd Annual Conference on Systems Engineering Research (CSER 2026)},
month = {apr},
year = {2026},
url = {https://mediatum.ub.tum.de/1851162},
abstract = {A potential application of LLMs in SE is the production of sets of requirements for a particular problem. In this paper, we survey how researchers evaluate the quality of requirements generated by LLMs and assess the adequacy of such approaches. Particularly, we show that metrics traditionally employed to evaluate the similarity of LLM-generated artifacts against ground truths do not perform well for requirements. The underlying reason is that similarity ought not to be achieved on the textual output but on the problem space captured by the requirements.},
keywords = {requirements, large language models (LLM), CoPilot, semantic similarity, cosine similarity},
file = {2026/2026_GarciaAlarcia_ExploringSimilarityAssessmentsLLMRequirements.pdf}
}