New capabilities for automated code generation have been introduced with the integration of Large Language Models (LLMs) into software engineering. However, as LLMs often generate code with hidden security vulnerabilities or maintainability issues, the reliability of these models remains a concern. The lack of robust evaluation methods is a major obstacle as traditional human review is not scalable, while current automated metrics such as pass@k fail to assess deep code quality dimensions. Existing evaluation benchmarks fail to represent the context of professional real-world software development, leading to a gap between measured and actual model performance.

This thesis aims at establishing a scalable evaluation methodology for LLM-based code assistance systems. It investigates the use of the LLM-as-a-Judge paradigm to assess code quality. To test the feasibility of this approach, a ground truth derived from human experts or static analysis tools will be compared to the judgments.

The objective of this thesis is to create an approach to solve the problem of limited scalability and overly shallow evaluation of AI-generated code. This includes the analysis of the state of the art regarding LLM code generation benchmarks as well as the demonstration of the solution by an experimental study. The thesis must include a suitable experimental evaluation based on a comparison of the evaluation outcomes of the proposed method against human experts.