Thinking Machines Lab

Debug root causes across the kernel, NCCL, scheduler, application, and telemetry layers.Provide embedded, hands-on support during critical training runs and major incidents through resolution.Act as the primary technical contact for researchers when infrastructure problems have unclear ownership.Lead postmortems and build tooling to prevent recurring incidents.Mentor engineers in cr

Own the reliability, performance, and uptime of large-scale post-training and reinforcement learning jobs.Partner with research teams during active model runs to debug failures and accelerate iteration.Debug distributed-system failures across accelerators, networking, storage, schedulers, and training frameworks.Build monitoring, alerting, automated recovery, checkpointing, fault to

Design, build, and operate distributed systems for scheduling, resource orchestration, checkpointing, and fault recovery across large multi-node GPU clusters.Own the reliability, latency, and throughput of live production systems serving concurrent training and sampling workloads.Implement graceful degradation and fast recovery from node failures, preemptions, and network partitions

Design, build, and maintain research infrastructure including evaluation frameworks, training systems, experiment-tracking platforms, and visualization tools.Develop end-to-end solutions across backend systems, data pipelines, and user-facing applications.Partner directly with researchers to identify bottlenecks, gather feedback, prioritize work, and measure adoption.Build reproduci

Design, build, and maintain a platform for authoring, running, tracking, and analyzing model evaluations.Develop evaluation libraries, distributed backend systems, data pipelines, APIs, and user-facing applications end to end.Build abstractions for evaluation tasks, environments, graders, datasets, and model outputs.Improve reproducibility and trust through versioning, provenance, o

Design, build, and ship full-stack web application features spanning backend services, APIs, and frontend interfaces.Prototype with emerging model capabilities and translate research results into product experiences.Take product ideas from rough concepts to live user-facing products and iterate based on feedback.Partner with researchers to understand model capabilities and constrain