Recursive improvement for AI systems

Datta builds the improvement loop for AI models and agents.

Get in touch Why →

Define success

The team defines success and the tests used to measure it. Part of the data is held back so nothing gets tuned on it.

What we sawEveryone measured the same problem with a different test, so nobody could compare results.

Run the baseline

Datta runs the current model on the test data the way it would run in production, and records every decision and when it was made.

What we sawBaselines were rebuilt by hand, and timing was measured after the fact.

Find what fails

Failures are grouped and compared with cases that worked. Each one is checked against its label, because the label is often what is wrong.

What we sawNobody checked whether two labelers agreed. Label mistakes showed up as model mistakes.

Choose the next experiment

Datta looks at the failures and the record of every past experiment, then picks the next thing to test. It may revise the training approach, fix labels or identify new data to collect.

What we sawExperiment history lived in notebooks and chat threads. People reran experiments that had already been run.

Run experiments in parallel

Changes to data, labels, training, agent instructions, tool-calling code and runtime settings run side by side, each with its own results. The held-out data is never touched.

What we sawExperiments ran one at a time, and the cheap fix was rarely tried first.

Measure and record

Every result is scored against the same tests. Datta records what each experiment changed and whether it helped. The winner becomes the new baseline.

What we sawOne accuracy number hid the tradeoff. In voice, faster usually means more people get cut off.

Why

We sat with researchers and asked why model releases took so long. Running experiments, testing the results, finding the right data and labeling it all took time. Experiments ran one at a time, and work was repeated when findings were not shared.

The steps live in different tools, owned by different people. An experiment can reveal a labeling problem. An underperforming agent can expose something the evaluation never tested. One bad data source can poison a training run. Someone still has to bring those findings together and decide what to do next.

We were doing parts of this loop by hand. Datta is the loop running on its own: find the failure, make a change, check whether it helped, repeat.

The loop

Six steps, one loop.

The team defines success and the tests used to measure it. Datta runs everything else and keeps the record.

Closing the loop

The next round starts from what worked.

Datta records what each experiment changed and whether it helped, then uses those results to choose the next experiment.

Get in touch Starting with voice →

First test case

We are starting with voice.

Our first test case is end-of-turn detection: knowing when a person has finished speaking. Voice agents either cut people off or wait too long. We have worked on voice products and know how many small failures sit behind that one problem, each with several possible causes. That is what the loop is for.

Our current prototype supports running experiments, comparing outputs and recording feedback. We are building the automated improvement loop around it. A researcher at UC San Diego is using it now.