Your browser does not support javascript! Please enable it, otherwise web will not work for you.

AI Evaluation Engineer (QA)

Home > Python programming jobs

AI Evaluation Engineer (QA) in USA

  • Appnovation Technologies
  • Full time
  • Email
  • New York, NY

Responsibilities

  • Run human and automated evaluations across large question sets, including hundreds of thousands or millions of evaluations.
  • Statistically measure factual grounding and before-and-after accuracy lift.
  • Build metrics frameworks that quantify answer quality improvements.
  • Design load and quality tests as the corpus scales.
  • Define and maintain test plans, test cases, and quality gates.
  • Automate regression and evaluation suites and integrate them into CI/CD.
  • Report quality metrics to technical and non-technical stakeholders.
  • Collaborate with engineering to reproduce, triage, and verify fixes.
  • Continuously improve QA processes and test coverage.

Requirements

  • Bachelor’s degree in a technical field or equivalent experience.
  • 4+ years of experience in QA or test engineering with exposure to data or ML systems.
  • Strong Python and data-science techniques for measuring factual grounding and answer quality.
  • Experience with LLM evaluation frameworks and statistical analysis.
  • Ability to build automated, large-scale evaluation harnesses and human-in-the-loop A/B tests.
  • Experience working across outputs from multiple LLM providers.
  • Experience with test automation frameworks and scripting.
  • Strong analytical, problem-solving, communication, and collaboration skills.
  • Prior consulting experience.
  • Prior Life Sciences industry experience and connections are preferred.

Benefits

  • Equal opportunity employer with a commitment to diversity, equity, and inclusion.
  • Accommodations are available upon request throughout the recruitment process.

Appnovation Technologies

Appnovation is a Global, Full-Service Digital Partner. We design, develop, and deliver digital experiences that drive measurable business transformation. With nearly 20 years of experience and thousands of successful projects delivered, we partner with industry leaders like Pfizer, Google, Yeti, ...

Similar positions

Software Engineer in Test

  • Roadie
  • Full time
  • USA
  • 09/13/2026
  • Salary: Competitive
  • Remote

Principal Software Engineer - Builder Experience

  • Elastic
  • Full time
  • Remote
  • 09/13/2026
  • Salary: €73k-€116k
  • Greece

Principal Software Engineer - Builder Experience

  • Elastic
  • Full time
  • Remote
  • 09/13/2026
  • Salary: zł 369k-zł 584k
  • Poland

Senior Devops Engineer

  • Woliba
  • Full time
  • USA
  • 09/13/2026
  • Remote

Senior Core Infrastructure Engineer

  • Oracle
  • Full time
  • USA
  • 09/13/2026
  • Salary: Competitive
  • Nashville, TN

Listings

by


Job Type

Language

Country

CodingJobBoard.com © 2026