AI Engineer, Post-Training & Evaluation

Title: AI Engineer, Post-Training & Evaluation

Location: San Francisco, CA, US

## **The role**

Own the models behind Coval’s judges. Coval is the evaluation platform for conversational AI: we help enterprise teams simulate, monitor and improve voice and chat agents before those agents ever reach a customer. Because our product is the judge, this role sits at the center of the company.

You’ll own the model side end to end. You’ll design and ship the LLM-as-a-judge evaluations that score voice and chat conversations at scale, then sharpen them with LoRA fine-tunes on open-weight models and non-tuning work like tool use and context engineering. Your improvements ship directly into the product enterprises rely on to launch dependable voice agents.

## **What you’ll do**

• Design and ship the LLM judges that score conversations: rubrics, judge prompting, and calibration against human raters
• Build the reliability tracking that keeps judges honest from release to release
• Run post-training where training beats prompting: LoRA adapters on open-weight models, trained on data distilled from our own scored conversations
• Everywhere else, do the non-tuning capability work: tool use, structured outputs, and context engineering
• Keep asking which lever actually moves the metric, and focus there

## **What we’re looking for**

• 4+ years as an engineer, with strong systems instincts
• A track record of shipping measurable quality gains by pairing rigorous measurement with targeted training, not vibes and prompt tweaks
• You’ve built eval infrastructure you actually trust
• You’ve run LoRA/SFT experiments end to end, including preparing your own data rather than relying on clean data from other teams, so you know which lever really wins
• High agency: on a small Series A team, nobody hands you tickets. You figure out what matters next and drive it there
• Ambitious and kind. You’ll be part of the group that founds our engineering org, sets its bar and grows the team