Your browser does not support javascript! Please enable it, otherwise web will not work for you.

AI Benchmark Engineer

Home > Python programming jobs

AI Benchmark Engineer in Remote

  • LILT (Production)
  • Full time
  • Email
  • United Arab Emirates

Responsibilities

  • Design and build Terminal-Bench tasks that evaluate coding agents on multilingual software challenges.
  • Create realistic Arabic-language datasets, files, prompts, and task environments without relying on English translations.
  • Develop reference implementations and deterministic verifier scripts, using rubric-based judging only when necessary.
  • Analyze execution logs and calibrate task difficulty from Easy to Very Hard across Haiku, Sonnet, and Opus model tiers.
  • Participate in creation, human review, calibration review, audit, and automated quality-control processes.
  • Ensure benchmark fairness, grammatical accuracy, multilingual robustness, and integrity.

Requirements

  • 5+ years of industry experience in software engineering.
  • A proven track record at leading technology companies and/or graduation from top-tier engineering universities.
  • Native or near-native Arabic fluency with deep knowledge of grammar, register, and phrasing, plus high English proficiency.
  • Strong proficiency in Python, standard shell scripting, and data processing.
  • Extensive experience with Terminal/CLI-based development workflows and familiarity with coding agents.
  • Deep understanding of encoding and decoding, Unicode normalization, locale-dependent conventions, text I/O, toolchain interoperability, and safe string operations.
  • For relevant language work, knowledge of bidirectional and RTL handling, font fallbacks, and rendering or typography in user interfaces or artifacts.
  • Reliable availability and commitment to completing accepted tasks on time.

Benefits

  • Remote freelance work with flexible scheduling and no fixed hours.
  • Competitive rates and prompt payments.
  • Access to diverse, innovative AI and language-technology projects and a global expert community.
  • Work availability fluctuates, hours are not guaranteed, and the role is intended as supplemental income rather than a primary full-time job.
  • Most tasks require at least 2 hours per day or 15–20 hours per week.
  • The role is a 1099 contractor position without health insurance, paid time off, or retirement contributions; contractors are responsible for their own taxes.
  • Geographic restrictions may apply for regions subject to international embargoes or sanctions.

Salary: Competitive

LILT (Production)

LILT builds an enterprise translation and localization platform that combines AI models with human linguists to produce multilingual content for businesses and governments. The company also provides data labeling and evaluation services for AI developers, plus managed translation services integra...

Similar positions

Senior Data Analytics Engineer

  • MariaDB plc
  • Full time
  • Remote
  • 10/02/2026
  • Salary: Competitive
  • India

Staff Engineer SDET

  • Aviatrix
  • Full time
  • USA
  • 10/02/2026
  • Salary: $189k - $223k/yr
  • Santa Clara, CA

Security Engineer

  • Deepgram
  • Full time
  • USA
  • 10/02/2026
  • Salary: $150k - $245k/yr
  • Remote

GTM AI Automation Engineer

  • Factored
  • Full time
  • Remote
  • 10/02/2026
  • Americas

Senior Site Reliability Engineer

  • Qutwo
  • Full time
  • Others
  • 10/02/2026
  • Helsinki, Finland

Listings

by


Job Type

Language

Country

CodingJobBoard.com © 2026