Your browser does not support javascript! Please enable it, otherwise web will not work for you.

Software Engineer, Training Infrastructure

Home > Python programming jobs

Software Engineer, Training Infrastructure in USA new

  • Thinking Machines Lab
  • Full time
  • Email
  • San Francisco, CA

Responsibilities

  • Design, build, and operate distributed training infrastructure for Tinker and internal frontier-model training, including scheduling, orchestration, checkpointing, and recovery.
  • Improve training throughput and cost efficiency through profiling, parallelism strategies, memory optimization, and kernel- and communication-level tuning.
  • Support new open-weights and in-house model architectures, including mixture-of-experts models.
  • Build automated failure detection, job recovery, multi-tenant isolation, and observability for long-running training jobs.
  • Support multi-tenant LoRA fine-tuning and tightly coupled sampling and training for reinforcement learning workloads.
  • Collaborate with Tinker API, frameworks, and product teams to maintain consistency between infrastructure and the user-facing SDK.

Requirements

  • 4+ years of software engineering experience, including building and operating production distributed systems.
  • Strong Python programming skills and experience with a systems language such as C++, Rust, or Go.
  • Hands-on experience with large-scale model training or inference infrastructure using frameworks such as PyTorch.
  • Experience debugging complex performance and reliability issues in distributed environments.
  • Preferred: experience training or serving large language models at scale, including mixture-of-experts architectures.
  • Preferred: experience with LoRA, parameter-efficient fine-tuning, or supervised and reinforcement learning pipelines for LLMs.
  • Preferred: experience with cluster orchestration and scheduling, such as Kubernetes or Slurm, and fault-tolerant checkpointing.
  • Preferred: experience building observability and profiling tooling.
  • Preferred: contributions to open-source ML systems projects such as PyTorch, Megatron, vLLM, or SGLang.

Benefits

  • Based in San Francisco, California or New York, New York.
  • Visa sponsorship is offered, subject to the stated qualification and visa-process caveat.
  • Generous health, dental, and vision benefits.
  • Unlimited PTO and paid parental leave.
  • Relocation support as needed.

Salary: $300k - $400k/yr

Thinking Machines Lab

Thinking Machines Lab pioneers advanced AI research, blending cutting‑edge theory with practical applications to solve complex problems across industries. Our interdisciplinary team pushes the boundaries of machine learning, neural architecture, and cognitive modeling to deliver innovative soluti...

Similar positions

Software Engineer, Datacenter Server Lifecycle

  • Anthropic
  • Full time
  • USA
  • 11/18/2026
  • Salary: $405k - $485k/yr
  • Remote

Solutions Engineer, AI Build Team

  • Stripe
  • Full time
  • USA
  • 11/18/2026
  • Salary: $240k-$360k
  • Remote

Infrastructure - Intermediate Software Engineer

  • Sony Pictures Imageworks
  • Full time
  • Canada
  • 11/18/2026
  • Vancouver

Senior Software Engineer, Simulator Platform

  • Waymo
  • Full time
  • USA
  • 11/18/2026
  • Salary: $213k - $263k/yr
  • Mountain View, CA

Associate Software Engineer

  • Sony Pictures Imageworks
  • Full time
  • Canada
  • 11/18/2026
  • Vancouver