Your browser does not support javascript! Please enable it, otherwise web will not work for you.

Member of Technical Staff, Evals

Home > Python programming jobs

Member of Technical Staff, Evals in USA new

  • Profound
  • Full time
  • Email
  • New York, NY

Responsibilities

  • Build Profound’s evaluation system with golden datasets, scorers, LLM judges, programmatic verifiers, regression alerts, and dashboards.
  • Create evaluations for brand extraction, visibility scoring, sentiment, FactCheck, and citation classification pipelines.
  • Define and measure AI Marketer quality across multi-step tool use, analytical correctness, and content quality.
  • Study answer-engine behavior across runs, platforms, regions, and personas using sound sampling and statistical methods.
  • Collect customer signals, cluster failure modes, and convert regressions into permanent guardrails.
  • Build debugging tools for model and agent behavior and create labeling workflows and annotation guidelines.
  • Integrate evaluations into CI, launch reviews, and model-selection processes.

Requirements

  • At least 3 years of experience building evaluation or measurement systems, including AI evaluations, experimentation, ranking, or search quality.
  • Strong Python and data skills with a record of shipping production systems.
  • Deep familiarity with LLM-as-judge evaluation, benchmark design, agent harnesses, calibration, and significance testing.
  • Strong experimental-design skills and the ability to separate signal from noise in non-deterministic systems.
  • Current understanding of model and agent behavior and related research.
  • Evidence of shipped benchmarks, datasets, open-source tools, or publications.
  • Ability to work in a fast-paced startup environment with rapid iteration.
  • Preferred: experience with search, retrieval, citation, or fact-verification systems.
  • Preferred: publications at NeurIPS, ICML, ICLR, ACL, or COLM.

Benefits

  • On-site work in the New York City or San Francisco office.
  • Base salary range of $215,000 to $300,000, plus equity and a full range of benefits and perks.

Salary: $215k - $300k/yr

Profound

Profound builds a SaaS marketing intelligence platform that helps enterprise brands measure and improve how they appear in AI-driven discovery across tools like ChatGPT and Perplexity. The product delivers real-time visibility analytics, competitive intelligence, and optimization insights for sea...

Similar positions

Software Engineer L5

  • Netflix
  • Full time
  • USA
  • 10/30/2026
  • Salary: $388k-$619k
  • Remote

Senior Software Engineer - MAAS

  • Canonical
  • Full time
  • Remote
  • 10/30/2026
  • EMEA, Americas

Software Engineer I

  • UiPath
  • Full time
  • Others
  • 10/30/2026
  • Bengaluru, India

Software Architect/Engineer 6

  • Netflix
  • Full time
  • USA
  • 10/30/2026
  • Salary: $499k-$820k
  • Remote

Senior Software Engineer

  • Mozilla
  • Full time
  • Remote
  • 10/30/2026
  • UK, Canada

Listings

by


Job Type

Language

Country

CodingJobBoard.com © 2026