Your browser does not support javascript! Please enable it, otherwise web will not work for you.

AI QA & Evaluation Engineer

Home > Python programming jobs

AI QA & Evaluation Engineer in Others

  • Elastic
  • Full time
  • Email
  • Bengaluru, India

Responsibilities

  • Design and implement comprehensive QA strategies for AI/ML systems, including accuracy, bias, robustness, regression, and integration testing.
  • Create self-contained evaluation tasks with prompts, supporting files, and detailed grading rubrics.
  • Automate validation suites for agentic and multi-agent systems, ML models, integration testing, and CI/CD pipelines.
  • Validate that AI/ML models consume accurate, authorized, properly structured, and high-quality data during training and inference.
  • Observe and document AI agent behavior, model performance, hallucinations, and data-grounded output accuracy.
  • Refine evaluation tasks and rubrics using feedback and collaboration to improve benchmarking methodologies.
  • Ensure AI data sources and structures meet security, privacy, governance, regulatory, and compliance standards.
  • Collaborate across IT, engineering, operations, data, integrations, PMO, CRM, risk, compliance, and business technology teams.
  • Monitor developments in AI and provide technical recommendations to the organization.

Requirements

  • Proficiency in Python, TypeScript, or other programming languages used in AI and test automation.
  • Experience designing or applying rubric-based evaluations, structured grading, or scoring frameworks.
  • Direct experience with LLM evaluation frameworks and benchmarking tools such as LangSmith and Confident AI.
  • Knowledge of the GenAI stack, including Retrieval Augmented Generation (RAG).
  • Experience with LLM solutions such as Azure OpenAI, Vertex AI, or ChatGPT Enterprise.
  • Experience with cloud platforms including Azure, GCP, or AWS.
  • Thorough understanding of DevOps, automation, and CI/CD tools including GitHub and Terraform.
  • Understanding of AI ethics, risk management, and data governance.
  • Strong attention to detail and ability to identify subtle data hallucinations, logic errors, and inconsistencies.
  • Advanced written communication skills for documenting nuanced observations and feedback.

Benefits

  • Competitive pay based on the work performed rather than previous salary.
  • Health coverage for employees and families in many locations.
  • Distributed work environment with flexible locations and schedules for many roles.
  • Generous vacation allowance.
  • Employer matching of up to $2000, or local currency equivalent, for financial donations and service.
  • Up to 40 hours per year for volunteer projects.
  • At least 16 weeks of parental leave.

Salary: Competitive

Elastic

Elastic is the leading platform for search-powered solutions, and Elastic helps everyone — organizations, their employees, and their customers — find what they need faster while keeping applications running smoothly, and protecting against cyber threats. When you tap into the power of Elastic Ent...

Similar positions

Software Engineer in Test

  • Roadie
  • Full time
  • USA
  • 09/13/2026
  • Salary: Competitive
  • Remote

Principal Software Engineer - Builder Experience

  • Elastic
  • Full time
  • Remote
  • 09/13/2026
  • Salary: €73k-€116k
  • Greece

Principal Software Engineer - Builder Experience

  • Elastic
  • Full time
  • Remote
  • 09/13/2026
  • Salary: zł 369k-zł 584k
  • Poland

Senior Devops Engineer

  • Woliba
  • Full time
  • USA
  • 09/13/2026
  • Remote

Senior Core Infrastructure Engineer

  • Oracle
  • Full time
  • USA
  • 09/13/2026
  • Salary: Competitive
  • Nashville, TN

Listings

by


Job Type

Language

Country

CodingJobBoard.com © 2026