Thesis: Human-Centered Evaluation of LLM-Based Systems at Work (Waldkirch (near Freiburg), Waldkirch (near Freiburg), Waldkirch (near Freiburg), Waldkirch )

Summer Semester 2027 – fixed term of 3–6 months

Do LLM-based and agentic AI systems actually improve the way we work, and what changes when they are introduced? In this thesis, you will develop and apply a human-centered approach to evaluating these systems in an industrial work environment. While technical evaluations can measure output quality, they do not show whether a system improves entire workflows, better supports employees, changes collaboration, or affects skills over time.

YOUR TASKS:

• Review and analyze relevant scientific literature to establish the theoretical and methodological foundations of a human-centered assessment framework.
• Define measurable indicators to assess the impact of an AI-supported workflow on individuals, teams, and the organization.
• Conduct an empirical study combining system usage data with methods such as task-based experiments, surveys, interviews, or observations.
• Develop recommendations for evaluating and deploying LLM-based systems in ways that create sustainable value while addressing risks such as over-reliance, skill erosion, and unclear accountability.

The expected outcome is a practical assessment framework and an evidence-based case study of an LLM-based system. You will examine not only whether the system improves immediate productivity, but also other selected dimensions, such as work quality, the effort required to verify results, employee capabilities, collaboration, and organizational processes. A critical examination of the underlying assumptions, limitations, and potential failure scenarios will be an essential part of the thesis.

YOUR PROFILE:

• You are currently pursuing an advanced master's degree in Human-Computer Interaction, Computer Science, Psychology, Cognitive Science, Information Systems, or a related field.
• You have a good understanding of empirical research methods. Experience with interviews, surveys, user studies, experimental design, or quantitative data analysis is an advantage. Familiarity with LLM-based systems, agentic AI, or workplace technologies is also beneficial.
• You are interested in formulating research questions, combining qualitative and quantitative evidence, and critically distinguishing short-term productivity gains from longer-term changes in skills, collaboration, and work practices.
• You can quickly grasp complex technical challenges, work independently and systematically, and communicate your findings clearly within an interdisciplinary team.
• You have a strong interest in human-centered AI, AI ethics, the future of work, and the responsible deployment of emerging technologies.
• You have good written and spoken English skills. German is an advantage, particularly for conducting interviews with employees.

You don't meet every requirement yet? We still encourage you to apply. We value curiosity, a strong willingness to learn, and the ability to approach complex problems systematically. You can also develop relevant skills during your thesis.

Contact: Sarah Disch