Your browser does not support javascript! Please enable it, otherwise web will not work for you.

Machine Learning Training Framework Engineer

Home > Python programming jobs

Machine Learning Training Framework Engineer in USA new

  • ByteDance
  • Full time
  • Email
  • San Jose, CA

Responsibilities

  • Develop training and inference frameworks for large-scale recommendation models and large language models.
  • Extend open-source models and frameworks to support business requirements and improve framework architecture.
  • Optimize coordination among training, generation, reward computation, and evaluation, including model synchronization and data flow.
  • Improve distributed execution, parallelism, GPU memory management, tiered storage, data loading, resource scheduling, throughput, utilization, and cost efficiency.
  • Deliver system designs and implementations through model-quality validation, performance testing, and production deployment.
  • Build testing, benchmarking, observability, fault diagnosis, checkpointing, and recovery capabilities for reliable training and inference systems.
  • Track relevant technologies, adopt open-source developments, and contribute to the community where appropriate.

Requirements

  • Background in computer science, software engineering, artificial intelligence, or a related field, with foundations in data structures, algorithms, operating systems, and computer architecture.
  • Proficiency in Python or C++, with experience developing and debugging software in Linux environments.
  • Experience with code design, testing, software engineering, and technical problem solving.
  • Familiarity with at least one deep learning, recommendation-model training, or large-model post-training and inference framework, including framework development, model integration, or system optimization.
  • Understanding of distributed systems or parallel computing fundamentals for multi-GPU training or inference.
  • Ability to use code analysis, profiling, and experiments to identify bottlenecks and validate improvements.
  • Ability to collaborate with modeling and infrastructure teams and balance efficiency, reliability, development cost, and long-term architecture.
  • Preferred experience developing core training, post-training, or inference framework components and contributing to open-source projects.
  • Preferred experience adapting open-source models or frameworks for production, including integration, functionality extensions, performance tuning, compatibility management, or version upgrades.
  • Preferred experience with recommendation, advertising, or search training systems, sparse features and embeddings, parameter servers, long-sequence training, or online learning.
  • Preferred experience with large-model post-training systems involving SFT, RL, OPD, training-generation coordination, reward computation, synchronization, or asynchronous execution.
  • Preferred experience optimizing GPU workloads through CUDA, Triton, kernel development, memory optimization, high-performance communication, or distributed parallelism.
  • Preferred experience with reliability or performance engineering for large-scale training or inference systems, including checkpointing, recovery, elastic execution, observability, tiered storage, scheduling, or benchmarking.

ByteDance

ByteDance is a global incubator of platforms at the cutting edge of commerce, content, entertainment and enterprise services - over 2.5bn people interact with ByteDance products including TikTok. Creation is the core of ByteDance's purpose. Our products are built to help imaginations thrive. This...

Similar positions

Bitcoin Security Engineer

  • MicroStrategy
  • Full time
  • USA
  • 11/21/2026
  • Salary: $104k - $187k/yr
  • Tysons, VA

Software Engineer I, Backend

  • Affirm
  • Full time
  • Canada
  • 11/21/2026
  • Remote

Senior Full Stack Engineer

  • Pavago
  • Full time
  • Remote
  • 11/21/2026
  • Jamaica

Lead Cloud Infrastructure Engineer (GCP)

  • Egen
  • Full time
  • Others
  • 11/21/2026
  • Hyderābād, India

Lead DevOps Engineer

  • L3Harris
  • Full time
  • USA
  • 11/21/2026
  • Salt Lake City, UT