Your browser does not support javascript! Please enable it, otherwise web will not work for you.

QA Automation Engineer - AI

Home > Python programming jobs

QA Automation Engineer - AI in USA new

  • Anaplan
  • Full time
  • Email
  • Remote

Responsibilities

  • Develop and execute end-to-end automated test coverage for GenAI evaluations, APIs, UIs, and performance tests using Python.
  • Create testing strategies for conversational AI, agentic systems, and LLM-powered workflows.
  • Maintain automated prompt-testing and regression-test suites to detect unintended changes in model behavior.
  • Support evaluation frameworks measuring accuracy, relevance, safety, consistency, and latency.
  • Curate test datasets and golden examples representing diverse user scenarios and edge cases.
  • Assist with adversarial testing for hallucinations and bias and support production alerting integration.
  • Maintain internal testing frameworks, support developer validation and user acceptance testing, and document procedures and known issues.

Requirements

  • Solid QA or test engineering experience with exposure to or interest in AI/ML systems.
  • Hands-on Python coding experience.
  • Understanding of GenAI technologies, LLMs, prompt engineering, and AI application patterns.
  • Experience with test automation frameworks and scripting, including Python, Pytest, API testing, and UI testing.
  • Knowledge of functional, integration, regression, performance, and security testing.
  • Ability to evaluate test criteria for non-deterministic systems.
  • Experience with API testing tools such as Postman and REST Assured and with backend testing.
  • Familiarity with CI/CD pipelines and automated testing integration.

Anaplan

Anaplan builds a cloud platform for enterprise planning, forecasting, and scenario modeling used by finance, supply chain, sales, and workforce teams. It sells subscription software and related services to large organizations that need connected, cross-functional planning. Founded in 2006 and hea...

Similar positions

Staff Machine Learning Engineer

  • Altana
  • Full time
  • USA
  • 10/15/2026
  • Salary: $205k - $240k/yr
  • Boston, MA

Security Platform Automation Engineer

  • Apple
  • Full time
  • USA
  • 10/15/2026
  • Seattle, WA

Analytics Engineer

  • Reddit
  • Full time
  • USA
  • 10/15/2026
  • Salary: $164k-$230k
  • Remote

Senior AI Engineer

  • Infiterra S.A
  • Full time
  • Remote
  • 10/15/2026
  • Greece, UK

Security Operations Engineer (SecOps)

  • Collibra
  • Full time
  • USA
  • 10/15/2026
  • Salary: $116k - $145k/yr
  • Remote

Listings

by


Job Type

Language

Country

CodingJobBoard.com © 2026