Skip to main content
TALENT247 // SPECIALIST AI EXECUTIVE SEARCH & TECHNICAL RECRUITMENT
OPS · MLOps & Platform

Staff ML Platform & Cluster Reliability Engineer

San Francisco, CA · Full-time · Hybrid

AI Specialism

MLOps & Platform

Workplace

Hybrid

Location

San Francisco, CA

Compensation

$260k – $350k

annual package

Role Dossier & Specification

REF: STAFF-ML-PLATFORM-

THE CLIENT & MANDATE

Our client is a premier San Francisco frontier AI laboratory operating a planet-scale heterogeneous compute fleet spanning hundreds of thousands of GPUs and custom AI accelerators across multiple cloud providers and bare-metal datacenters. Keeping training superclusters and global low-latency inference regions operating as a single, self-healing compute fabric is essential to both research velocity and commercial API reliability.

They have partnered with Talent247 to appoint a Staff ML Platform & Cluster Reliability Engineer in San Francisco, CA. Acting as a technical anchor across cluster orchestration, fleet health automation, and storage streaming, the successful candidate will build the Go and Rust control planes that maximize goodput across billions of dollars of AI compute.

KEY RESPONSIBILITIES

  • Architect topology-aware Kubernetes gang schedulers and multi-cluster workload placement engines optimizing NVLink/InfiniBand locality across pre-training, RL rollout, and global inference fleets.
  • Build autonomous hardware validation, continuous active/passive health probing, and automated node-quarantine/drain pipelines eliminating GPU Xid faults, flapping optical links, and silent compute corruption without human paging.
  • Design high-throughput petabyte checkpoint and model-weight distribution fabrics in Rust and Go capable of saturating 400G/800G NICs from object storage into GPU memory in seconds.
  • Establish fleet-wide goodput accounting, preemption-aware job elasticity, and unified eBPF/OpenTelemetry observability correlating kernel/driver telemetry with job-level step times.
  • Lead multi-region disaster-recovery exercises, capacity ramp-ups for newly delivered 10,000+ accelerator clusters, and blameless architectural post-mortems.
  • Mentor senior infrastructure engineers while partnering closely with distributed training leads and cloud/hardware vendor engineering teams.

TECHNICAL ENVIRONMENT & OFFERING

Stack: Go, Rust, Python, Kubernetes (Kueue / custom operators & CNI/DRA plugins), Linux kernel/eBPF, InfiniBand/RoCEv2, Prometheus/ClickHouse, multi-cloud & bare-metal GPU/ASIC clusters. Workplace: Hybrid in San Francisco, CA. Compensation: $260,000 – $350,000 base salary plus flagship frontier-lab equity and comprehensive US benefits.

Core Requirements

ESSENTIAL QUALIFICATIONS

  • 7+ years of infrastructure, distributed systems, or production SRE/platform engineering experience, including 3+ years operating large-scale GPU or AI accelerator clusters.
  • Expert-level systems development in Go or Rust alongside deep mastery of Kubernetes internals (custom schedulers, CRDs/operators, Dynamic Resource Allocation, device plugins).
  • Deep Linux systems troubleshooting expertise across PCIe/NVLink topologies, GPU drivers/DCGM, RDMA/RoCEv2/InfiniBand networking, and kernel memory/IO subsystems.
  • Proven track record improving cluster utilization/goodput and building automated self-healing remediation at multi-thousand-node scale.
  • Calm technical leadership across high-stakes cluster bring-ups and production reliability initiatives.

PREFERRED / BONUS EXPERIENCE

  • Experience building P2P or RDMA-accelerated container/model-weight distribution systems across thousands of GPU nodes.
  • Familiarity with both synchronous pre-training gang scheduling and elastic asynchronous RL actor-learner cluster workloads.
  • Core contributions to Kubernetes WG-Serving / Kueue, Volcano, Ray on K8s, or Linux eBPF observability tools.

Position Telemetry

Location San Francisco, CA (Hybrid)
Department MLOps & Platform
Employment type Full-time
Workplace type Hybrid
Salary range
$260k – $350k annual package