AI Infrastructure
New York, NY
On-site · Full-time
$225k – $320k
annual package
San Francisco, CA · Full-time · On-site
AI Infrastructure
On-site
San Francisco, CA
$290k – $385k
annual package
Our client is a premier San Francisco frontier AI laboratory operating some of the largest single-job GPU training supercomputers ever assembled. At tens of thousands of interconnected accelerators, hardware failures, silent data corruption, and network congestion become continuous statistical certainties rather than rare anomalies—making distributed training systems engineering a decisive moat for frontier model scaling.
They have partnered with Talent247 to hire a Staff Distributed Training Systems Engineer onsite in San Francisco, CA. Serving as a technical lead within their Compute & Training Foundations pillar, the appointed engineer will architect elastic fault-tolerant runtime primitives, collective communication schedules, and zero-stall checkpointing systems sustaining >95% effective cluster goodput across multi-month pre-training and RL runs.
Stack: C++20, CUDA, Python, PyTorch internals (c10d / TorchDistributed), NCCL, InfiniBand/RoCEv2 GPUDirect RDMA, Kubernetes/custom gang schedulers, Hopper & Blackwell superclusters. Workplace: Onsite in San Francisco, CA. Compensation: $290,000 – $385,000 base salary plus flagship equity grants, comprehensive healthcare, and full relocation support.