---
title: "Staff Distributed Training Systems Engineer"
slug: "staff-distributed-training-systems-engineer"
id: "e72b3c09-7961-4924-81dc-2581a3022687"
department: "AI Infrastructure"
department_code: "INF"
location: "San Francisco, CA"
workplace_type: "On-site"
employment_type: "Full-time"
compensation: "$290k – $385k (annual package)"
is_remote: false
is_featured: false
headcount: 1
canonical_url: "https://jobs.talent247.co/jobs/staff-distributed-training-systems-engineer"
markdown_url: "https://jobs.talent247.co/jobs/staff-distributed-training-systems-engineer.md"
apply_endpoint: "https://jobs.talent247.co/api/public/jobs/staff-distributed-training-systems-engineer/apply"
schema_endpoint: "https://jobs.talent247.co/api/public/jobs/staff-distributed-training-systems-engineer/form-schema"
webmcp_tool: "apply_for_role"
updated_at: "2026-11-15T17:37:18.217Z"
content_signal: "search=yes, ai-input=yes, ai-train=no"
---

# Staff Distributed Training Systems Engineer

## Position Telemetry

- **Reference:** `STAFF-DISTRIBUTED-`
- **Practice / Sector:** AI Infrastructure (`INF`)
- **Location:** San Francisco, CA
- **Workplace Type:** On-site
- **Employment Type:** Full-time
- **Compensation:** $290k – $385k (annual package)
- **Canonical HTML URL:** https://jobs.talent247.co/jobs/staff-distributed-training-systems-engineer

## Role Dossier & Specification

THE CLIENT & MANDATE
Our client is a premier San Francisco frontier AI laboratory operating some of the largest single-job GPU training supercomputers ever assembled. At tens of thousands of interconnected accelerators, hardware failures, silent data corruption, and network congestion become continuous statistical certainties rather than rare anomalies—making distributed training systems engineering a decisive moat for frontier model scaling.

They have partnered with Talent247 to hire a Staff Distributed Training Systems Engineer onsite in San Francisco, CA. Serving as a technical lead within their Compute & Training Foundations pillar, the appointed engineer will architect elastic fault-tolerant runtime primitives, collective communication schedules, and zero-stall checkpointing systems sustaining >95% effective cluster goodput across multi-month pre-training and RL runs.

KEY RESPONSIBILITIES
• Architect 5D-parallel distributed training runtimes (data, tensor, pipeline, context, and expert parallelism) scaling cleanly across 10,000 to 100,000+ next-generation GPUs.
• Design sub-minute fault detection, dynamic rank reconfiguration, and in-memory/NVMe asynchronous checkpointing engines that recover from GPU or link failures without full-job restarts.
• Co-optimize NCCL collective communication schedules, computation-communication overlap kernels, and topology-aware job placement over 400G/800G InfiniBand and RoCEv2 fabrics.
• Build deterministic silent-data-corruption (SDC) detectors, deterministic replay profilers, and per-iteration straggler attribution pipelines catching degraded silicon in flight.
• Collaborate directly with model architecture researchers to make novel sparse MoE routing, ultra-long-context attention, and FP8/FP4 low-precision training recipes hardware-optimal from day one.
• Mentor senior distributed systems and GPU kernel engineers across training runtime, cluster scheduling, and networking squads.

TECHNICAL ENVIRONMENT & OFFERING
Stack: C++20, CUDA, Python, PyTorch internals (c10d / TorchDistributed), NCCL, InfiniBand/RoCEv2 GPUDirect RDMA, Kubernetes/custom gang schedulers, Hopper & Blackwell superclusters. Workplace: Onsite in San Francisco, CA. Compensation: $290,000 – $385,000 base salary plus flagship equity grants, comprehensive healthcare, and full relocation support.

## Core Requirements

ESSENTIAL QUALIFICATIONS
• 7+ years of systems engineering experience in distributed ML training infrastructure, HPC supercomputing runtimes, or GPU collective communications.
• Deep production expertise in PyTorch distributed internals, Megatron-Core/FSDP, C++17/20, CUDA streams/events, and NCCL / RDMA networking mechanics.
• Proven track record diagnosing stragglers, deadlocks, memory fragmentation, and numerical divergence on multi-hundred- to multi-thousand-GPU training jobs.
• Strong command of GPU microarchitecture, HBM/NVLink/NVSwitch bandwidth hierarchies, and PCIe/NIC topology alignment.
• Demonstrated technical leadership driving cross-functional architecture across research, kernel, networking, and cluster operations teams.

PREFERRED / BONUS EXPERIENCE
• Direct experience operating or writing training runtimes on 4,096+ H100/B200 GPU clusters.
• Contributions to NCCL, PyTorch Distributed, Megatron-LM, DeepSpeed, TorchTitan, or custom RDMA transport libraries.
• Experience with asynchronous/elastic RL training architectures or disaggregated pre-training storage fabrics.

## Application Schema & Direct Apply (`apply_for_role`)

Candidates and authorized AI agents on behalf of a candidate can submit an application via the declarative WebMCP form tool `apply_for_role` on https://jobs.talent247.co/jobs/staff-distributed-training-systems-engineer or via HTTP `POST https://jobs.talent247.co/api/public/jobs/staff-distributed-training-systems-engineer/apply` (`application/json` or `multipart/form-data`). Non-browser HTTP agents must include header `X-Talent247-Intake-Token` obtained from `GET https://jobs.talent247.co/api/public/intake-token` (or automatically returned in `GET https://jobs.talent247.co/api/public/jobs/staff-distributed-training-systems-engineer/form-schema`).

### Form Fields

- `full_name` (text, required): Full name
- `email` (email, required): Email
- `phone` (phone, optional): Phone
- `location` (text, optional): Location
- `linkedin_url` (url, optional): LinkedIn URL
- `cover_letter` (textarea, optional): Cover letter
- `consent` (checkbox, required): I consent to my data being processed for this application
- `cv_file` / `cv_base64` + `cv_filename`: Upload a CV in .pdf or .docx format (<= 10 MB) or provide an `https://` LinkedIn profile URL in `linkedin_url`.

### Alternative: Talent Pool Intake (`join_talent_pool`)

If this mandate is not an exact match, register once in the Talent247 Talent Pool at https://jobs.talent247.co/talent-pool (`https://jobs.talent247.co/talent-pool.md`) via WebMCP tool `join_talent_pool` or `POST https://jobs.talent247.co/api/public/talent-pool`.
