AI Agent Evaluation Engineer

Home > Python programming jobs

AI Agent Evaluation Engineer in USA new

Cresta
Full time
Email

USA
11/13/2026
Report

Remote

Responsibilities

Define and maintain evaluation rubrics covering agent adherence, conversation outcomes, and user experience.
Write annotation guidelines and failure taxonomies, run calibration sessions, and measure inter-rater agreement.
Review and annotate production conversations, classify failure modes, and prioritize them by severity and frequency.
Trace failures to likely causes and route actionable findings to the appropriate engineering or product owners.
Build and operate an evaluation pipeline combining automated checks, LLM-based judges, and human review.
Calibrate automated judges against human labels and design stratified sampling strategies across languages, intents, caller types, and audio conditions.
Build and maintain golden sets, regression suites, coverage maps, and evaluations for edge cases and high-stakes conversations.
Define quality and latency thresholds and report quality trends with appropriate uncertainty to stakeholders and customers.
Partner with engineering, product, and forward-deployed teams to turn evaluation findings into prioritized recommendations.
Document evaluation methodology and direct or conduct weekly conversation-listening and labeling work.

Requirements

3+ years of professional experience in software engineering, applied machine learning, or data-intensive research, including 1–2 years working hands-on with production LLM systems.
Bachelor’s or Master’s degree in Computer Science, Computational Linguistics, Cognitive Science, Statistics, or a related field, or equivalent practical experience.
Experience building and operating an end-to-end evaluation system for a production LLM product, including rubrics, human labels, automated judges, and regression suites.
Deep understanding of golden sets, judges, regression suites, calibration, and evaluation methodology.
Fluency with LLM agents, including prompts, tool calls, context handling, multi-turn state, and trace analysis.
Strong Python programming skills and experience with LLM APIs; able to independently build and ship evaluation tooling.
Ability to write consistent rubrics, measure reviewer agreement, and design tests from open-ended quality questions.
Willingness to perform hands-on conversation listening, labeling, and failure-taxonomy development, personally and through QA partners.
Professional fluency in at least one language other than English sufficient to evaluate multilingual agent behavior.
Strong written communication skills and ability to author operational guidelines.
Fine-tuning and machine-learning frameworks are not required.

Benefits

Comprehensive medical, dental, and vision coverage for employees and families.
Flexible paid time off.
Paid parental leave for all new parents.
Retirement savings plan.
Remote work setup budget.
Monthly wellness and communication stipend.
In-office meal program and commuter benefits for onsite employees.
The hiring process begins with a self-paced CoderPad technical assessment for qualified applicants.

Cresta

Cresta builds a platform that transforms contact centers by merging AI and human intelligence. Targeting customer service teams, it enables them to turn conversations into valuable insights, automate processes, and implement best practices, ultimately enhancing performance and efficiency.

Similar positions

Machine Learning Engineer, Ads Ranking

SmartNews
Full time

Others
11/13/2026

Shibuya, Japan

Full-Stack Engineer (Backend Leaning)

ElevenLabs
Full time

Remote
11/13/2026

UK, Bulgaria

AI/ML Engineer

Kaizen Labs
Full time

USA
11/13/2026
Salary: $240k - $280k/yr

New York, NY

Contract Senior Data Engineer

Yld
Full time

Remote
11/13/2026

Europe

Data Engineer

SmartRent
Full time

USA
11/13/2026

Phoenix, AZ