Senior AI QA Automation Engineer

Home > Python programming jobs

Senior AI QA Automation Engineer in Remote new

Encora
Full time
Email

Remote
10/13/2026
Report

Bolivia

Responsibilities

Build automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, workflows, and model outputs.
Create benchmark datasets, golden responses, regression suites, evaluation metrics, and measurable release thresholds.
Test AI behaviors including hallucinations, grounding, citation accuracy, prompt injection, tool selection, and multi-agent coordination.
Develop agent-powered test-case generation, API and workflow validation, intelligent UAT, and UI/API/integration/performance/end-to-end automation.
Implement CI/CD quality gates and monitor production AI quality, drift, reliability, and recurring failure patterns.
Translate business requirements, workflows, and AI behaviors into measurable quality criteria.

Requirements

At least 5 years of experience in Software Quality Engineering, Test Automation, or Reliability Engineering.
Strong Python skills and experience building reusable automated testing frameworks.
Hands-on experience with Playwright and/or Selenium.
Strong experience with API, UI, integration, regression, end-to-end, and CI/CD-based test automation.
Experience with Azure AI Foundry Evaluations, RAG evaluation, and prompt testing.
Experience defining evaluation metrics, benchmark datasets, regression suites, and measurable quality thresholds.
Knowledge of hallucination testing, grounding, citation accuracy, prompt injection, and agent behavior validation.
Experience implementing CI/CD quality gates and production monitoring.
Bachelor’s degree in Computer Science, Software Engineering, Information Systems, or a related field.
Preferred experience testing Generative AI, RAG solutions, copilots, or AI agents.
Preferred experience with LLM evaluation frameworks, advanced prompt testing, adversarial testing, and Responsible AI testing.
Preferred experience developing agent-powered test generation, workflow validation, intelligent UAT assistants, and monitoring AI quality in production.

Benefits

Remote work arrangement.
Role available in Costa Rica, Peru, Colombia, and Bolivia.

Encora

Encora provides software and digital product engineering services for technology companies and enterprises, including cloud-native development, data engineering, and QA. It operates a global delivery model and supports outsourced product development and managed engineering teams. Headquartered in...

Similar positions

Security Platform Automation Engineer

Apple
Full time

USA
10/15/2026

Seattle, WA

Analytics Engineer

Reddit
Full time

USA
10/15/2026
Salary: $164k-$230k

Remote

Senior AI Engineer

Infiterra S.A
Full time

Remote
10/15/2026

Greece, UK

Security Operations Engineer (SecOps)

Collibra
Full time

USA
10/15/2026
Salary: $116k - $145k/yr

Remote

Accounting Automation Engineer

Ruby Labs
Full time

Remote
10/15/2026

EMEA