Your browser does not support javascript! Please enable it, otherwise web will not work for you.

Software Engineer - Benchmarking

Home > Python programming jobs

Software Engineer - Benchmarking in Remote

  • Office Hours
  • Full time
  • Email
  • Worldwide

Responsibilities

  • Prepare, clean, convert, validate, and maintain benchmark datasets and flag ambiguities that could compromise results.
  • Build infrastructure and pipelines that run evaluations consistently across model APIs and terminal agents.
  • Create lightweight containerized evaluation environments and viewers for tasking and tool-use assessment.
  • Support fine-tuning experiments and compare baseline with post-training model performance.
  • Build scoreboards and leaderboards showing model-, benchmark-, task-, domain-, and rubric-level results.
  • Develop tools to identify failure modes, compare models and agent scaffolds, and track capability changes over time.
  • Build HTML viewers and internal tools for task authoring, expert annotation, review, quality control, and structured data collection.
  • Collaborate with researchers and engineers to ensure evaluation data and published outputs are accurate, consistent, and integrated.

Requirements

  • 4+ years of professional experience building and maintaining complex systems.
  • Strong Python engineering skills and experience working in existing codebases and infrastructure.
  • Experience preparing, cleaning, validating, and maintaining datasets.
  • Experience with Docker and reproducible execution environments.
  • Ability to collaborate with researchers and scientists and translate methodology into working systems.
  • Hands-on AI evaluation experience or experience with Harbor, Terminal-Bench, or Inspect is a strong plus.
  • Experience fine-tuning or post-training open-source LLMs, agentic or tool-use evaluation, LLM-as-judge or rubric-based grading, data-heavy dashboards, leaderboards, visualizations, or related open-source or published benchmark work is preferred.

Benefits

  • Medical, dental, and vision coverage.
  • 401(k).
  • Monthly wellness and fitness stipend.
  • Paid time off and company holidays.
  • Annual company off-sites in Tahoe, Mendocino, Mexico City, San Diego, and Park City.
  • Parent-friendly policies, paid family leave, and remote flexibility.
  • Fully remote or hybrid work from the San Francisco or New York City offices.

Salary: $160k - $210k/yr

Office Hours

Office Hours is an on-demand expert network connecting leading organizations with the expertise they need, across industries, functions, and hard-to-reach domains. We believe human knowledge is the world's most valuable asset. Yet, even in an increasingly connected world, most of what people know...

Similar positions

Research Scientist - Machine Learning

  • ISEE
  • Full time
  • USA
  • 09/13/2026
  • Remote

Software Engineer in Test

  • Roadie
  • Full time
  • USA
  • 09/13/2026
  • Salary: Competitive
  • Remote

Principal Software Engineer - Builder Experience

  • Elastic
  • Full time
  • Remote
  • 09/13/2026
  • Salary: €73k-€116k
  • Greece

Principal Software Engineer - Builder Experience

  • Elastic
  • Full time
  • Remote
  • 09/13/2026
  • Salary: zł 369k-zł 584k
  • Poland

Senior Devops Engineer

  • Woliba
  • Full time
  • USA
  • 09/13/2026
  • Remote

Listings

by


Job Type

Language

Country

CodingJobBoard.com © 2026