AI Agent Evaluation Engineer

About the role:

What You'll Do :

Evaluation Design & Quality Definition

• Define and maintain the evaluation rubric for the agent across three axes: adherence (did it follow its design, call the right tools, respect guardrails), outcome (did the conversation achieve its goal), and conversation experience (latency, turn-taking, tone, repetition).
• Write the annotation guidelines and failure taxonomy the team uses to talk about failures precisely; run calibration sessions and measure inter-rater agreement.
  • Classify and prioritize failure modes by severity and frequency so engineering and product work on what matters most.

Conversation Analysis & Annotation

• Review production conversations by hand every week, leverage QA partners; open-code failures, cluster them, and turn the clusters into the evaluations worth automating.
• Trace each failure to its likely cause (prompt, knowledge gap, tool or flow bug, speech recognition) and route it to the right owner with enough evidence to act on.
• Evaluate across languages and report quality by segment (language, accent, audio quality, intent), not just in aggregate.

Scalable Evaluation Systems

• Build and operate the evaluation pipeline that scores every production conversation, combining automated checks, LLM-based judges, and human review.
• Calibrate every automated judge against human labels before trusting it; treat judge error as the primary measurement risk.
• Design sampling strategies (stratified by intent, language, caller type, audio conditions) so that reported changes in quality are real.
• Build and maintain the golden set and regression suite that every prompt, model, knowledge, and flow change is validated against before release.

Coverage & Edge Cases

• Maintain the failure-mode coverage map: which behaviors are tested, partially tested, or unmeasured; close the gaps.
• Design evaluations for behaviors no standard harness covers: high-stakes conversation types such as handoffs to a human, misuse and unwanted-caller handling, and resilience under upstream failures.
• Define the quality and latency thresholds the agent is held to.

Reporting & Cross-Functional Collaboration

• Report quality trends to internal stakeholders and customers clearly, with uncertainty stated and without burying bad news.
• Partner with forward-deployed engineers, backend engineers, product managers, and the voice/language engineering team to turn findings into prioritized recommendations.
• Document methodology so results are reproducible and the evaluation practice outlives any one person.

What We're Looking For:

• 3+ years of professional experience in software engineering, applied ML, or data-intensive research roles, including at least 1–2 years working hands-on with LLM-based systems in production.
• Bachelor’s or Master’s degree in Computer Science, Computational Linguistics, Cognitive Science, Statistics, or a related field, or equivalent practical experience.
• Built and operated an end-to-end evaluation system for a production LLM product (agent or otherwise): rubric, human labels, automated judge, regression suite.
• Deep understanding of evaluation methodology: what golden sets, judges, regression suites, and calibration are for, and when each fails. Tool-agnostic; you will work in our in-house tooling.
• Fluent in how LLM agents work: prompts, tool calls, context handling, multi-turn state. You can read a trace and tell what the agent was doing and why.
• Strong programming skills in Python and experience with LLM APIs; able to build and ship evaluation tooling independently. Fine-tuning and ML frameworks are not required.
• Able to write a rubric that independent reviewers would score the same way, and to measure whether they do.
• Able to design a test from an open-ended quality question: you decide what to measure and how, rather than executing a predefined plan.
• Willing to do non-scalable work weekly, both yourself and by directing our QA partners: listening to calls, labeling conversations, building a failure taxonomy from scratch.
• Professional fluency in at least one language other than English, sufficient to evaluate multilingual agent behavior directly.
• Strong written communication; you author the guidelines others follow.

Note on Our Hiring Process

We highly value your time and engineering craftsmanship. To ensure a streamlined and objective process, the first step for qualified applicants following an initial resume review is a self-paced CoderPad technical assessment. This allows you to showcase your practical coding and problem-solving skills in a real-world environment before moving into conversational interview rounds.

Perks & Benefits:

We offer a comprehensive and people-first benefits package to support you at work and in life:

• Comprehensive medical, dental, and vision coverage with plans to fit you and your family
• Flexible PTO to take the time you need, when you need it
• Paid parental leave for all new parents welcoming a new child
• Retirement savings plan to help you plan for the future
• Remote work setup budget to help you create a productive home office
• Monthly wellness and communication stipend to keep you connected and balanced
• In-office meal program and commuter benefits provided for onsite employees

Compensation at Cresta: 

Cresta’s approach to compensation is simple: recognize impact, reward excellence, and invest in our people. We offer competitive, location-based pay that reflects the market and what each individual brings to the table.

The posted base salary range represents what we expect to pay for this role in a given location. Final offers are shaped by factors like experience, skills, education, and geography. In addition to base pay, total compensation includes equity and a comprehensive benefits package for you and your family.