Evaluating AI-powered speech documentation for dental practices

Project overview

• Client: DAMPSOFT, market leader in dental practice management software in Germany

• Project goal: Build an evaluation framework for quality assurance and co-develop the AI documentation assistant AIVA

• Team: two TNG Consultants, working together with a three-person AI team from the client

• Project start: January 2026

• Key achievements: Release of the documentation assistant AIVA in July 2026, measurable quality improvement of several percentage points through systematic evaluation

The situation

DAMPSOFT develops practice management software for dental practices and is the German market leader with approximately 36 percent market share. Recently, an increasing number of start-ups emerged, aiming to transform individual sub-processes using AI-based approaches.

When we joined the project at the end of 2025, a first draft of the AI service "AIVA" already existed. The first feature, the automated transcription of patient conversations and creation of documentation for the medical patient record, had been developed and was in a testing phase with initial dental practices. A first evaluation experiment had been conducted, but there was no systematic and robust quality measurement yet.

Our target

Our first task was to design and build an evaluation framework that allows our client to evaluate the quality of the existing AI service in a structured way and to iteratively improve it. This has already been successfully completed and is now being actively used.

Currently, we are working as developers for the AI service itself, adding new features and implementing improvements. The client gives us considerable freedom: we are able to work exploratively and without rigid specifications, including the choice of AI technologies.

Our approach

At the beginning, we investigated how the quality of AI-generated conversation documentation in a patient record can be assessed. Building on this, we designed, tested and implemented an evaluation framework, comprising:

• a realistic test dataset, including feedback from dental professionals

• various metrics to measure the documentation quality

• automation, visualization, and tracing for repeatable and comprehensible evaluation runs.

The evaluation of the documentation against these metrics is AI-based, using LLMs, and is implemented in Python.

Once the evaluation framework was in place, we increasingly moved into co-developing the AI service itself, improving existing features and developing new functionality.

Our team of two TNG consultants works in close coordination with the client's technical lead, product owner, and one developer colleague. The project follows a dynamic, exploratory approach without fixed sprints, with a weekly alignment and ad-hoc coordination with the client as needed. This flexibility enables us to respond quickly to feedback from the testing phase.

A particular challenge was working with sensitive personal data: debugging issues from the test phase was complicated, as we could only reproduce problems abstractly rather than working with real examples from the data.

An important milestone was upgrading both the LLM (to GPT-5.4) and the speech-to-text model (to MAI-Transcribe-1.5), which was made possible by the structured evaluation and brought significant quality improvements.

The result

In early July 2026, the documentation assistant AIVA was released and integrated into DAMPSOFT's practice management systems DS-WIN and DS4. The assistant records patient conversations and creates documentation for patient records from them.

The evaluation framework developed for this purpose, including automation and visualization, enables our client to make changes to AIVA without compromising quality, while at the same time ensuring that its quality remains measurable. The quality of the service has improved by several percentage points, partly due to impulses from the evaluation framework.

Our client is very satisfied, and we continue to jointly develop the AI service together with DAMPSOFT. Examples of larger features that we are currently working on include enabling individualized documentation, as well as specifying appointment types.

We look forward to the continued collaboration and would also like to thank our colleagues Sebastian Zett and Julian Bollig, who presented this project jointly with DAMPSOFT at Big Techday 26.