---
title: "Staff ML Platform & Cluster Reliability Engineer"
slug: "staff-ml-platform-cluster-reliability-engineer"
id: "9258ec89-9a3c-48b8-b513-0e5e5c9b2dd5"
department: "MLOps & Platform"
department_code: "OPS"
location: "San Francisco, CA"
workplace_type: "Hybrid"
employment_type: "Full-time"
compensation: "$260k – $350k (annual package)"
is_remote: false
is_featured: false
headcount: 1
canonical_url: "https://jobs.talent247.co/jobs/staff-ml-platform-cluster-reliability-engineer"
markdown_url: "https://jobs.talent247.co/jobs/staff-ml-platform-cluster-reliability-engineer.md"
apply_endpoint: "https://jobs.talent247.co/api/public/jobs/staff-ml-platform-cluster-reliability-engineer/apply"
schema_endpoint: "https://jobs.talent247.co/api/public/jobs/staff-ml-platform-cluster-reliability-engineer/form-schema"
webmcp_tool: "apply_for_role"
updated_at: "2026-11-15T17:38:36.784Z"
content_signal: "search=yes, ai-input=yes, ai-train=no"
---

# Staff ML Platform & Cluster Reliability Engineer

## Position Telemetry

- **Reference:** `STAFF-ML-PLATFORM-`
- **Practice / Sector:** MLOps & Platform (`OPS`)
- **Location:** San Francisco, CA
- **Workplace Type:** Hybrid
- **Employment Type:** Full-time
- **Compensation:** $260k – $350k (annual package)
- **Canonical HTML URL:** https://jobs.talent247.co/jobs/staff-ml-platform-cluster-reliability-engineer

## Role Dossier & Specification

THE CLIENT & MANDATE
Our client is a premier San Francisco frontier AI laboratory operating a planet-scale heterogeneous compute fleet spanning hundreds of thousands of GPUs and custom AI accelerators across multiple cloud providers and bare-metal datacenters. Keeping training superclusters and global low-latency inference regions operating as a single, self-healing compute fabric is essential to both research velocity and commercial API reliability.

They have partnered with Talent247 to appoint a Staff ML Platform & Cluster Reliability Engineer in San Francisco, CA. Acting as a technical anchor across cluster orchestration, fleet health automation, and storage streaming, the successful candidate will build the Go and Rust control planes that maximize goodput across billions of dollars of AI compute.

KEY RESPONSIBILITIES
• Architect topology-aware Kubernetes gang schedulers and multi-cluster workload placement engines optimizing NVLink/InfiniBand locality across pre-training, RL rollout, and global inference fleets.
• Build autonomous hardware validation, continuous active/passive health probing, and automated node-quarantine/drain pipelines eliminating GPU Xid faults, flapping optical links, and silent compute corruption without human paging.
• Design high-throughput petabyte checkpoint and model-weight distribution fabrics in Rust and Go capable of saturating 400G/800G NICs from object storage into GPU memory in seconds.
• Establish fleet-wide goodput accounting, preemption-aware job elasticity, and unified eBPF/OpenTelemetry observability correlating kernel/driver telemetry with job-level step times.
• Lead multi-region disaster-recovery exercises, capacity ramp-ups for newly delivered 10,000+ accelerator clusters, and blameless architectural post-mortems.
• Mentor senior infrastructure engineers while partnering closely with distributed training leads and cloud/hardware vendor engineering teams.

TECHNICAL ENVIRONMENT & OFFERING
Stack: Go, Rust, Python, Kubernetes (Kueue / custom operators & CNI/DRA plugins), Linux kernel/eBPF, InfiniBand/RoCEv2, Prometheus/ClickHouse, multi-cloud & bare-metal GPU/ASIC clusters. Workplace: Hybrid in San Francisco, CA. Compensation: $260,000 – $350,000 base salary plus flagship frontier-lab equity and comprehensive US benefits.

## Core Requirements

ESSENTIAL QUALIFICATIONS
• 7+ years of infrastructure, distributed systems, or production SRE/platform engineering experience, including 3+ years operating large-scale GPU or AI accelerator clusters.
• Expert-level systems development in Go or Rust alongside deep mastery of Kubernetes internals (custom schedulers, CRDs/operators, Dynamic Resource Allocation, device plugins).
• Deep Linux systems troubleshooting expertise across PCIe/NVLink topologies, GPU drivers/DCGM, RDMA/RoCEv2/InfiniBand networking, and kernel memory/IO subsystems.
• Proven track record improving cluster utilization/goodput and building automated self-healing remediation at multi-thousand-node scale.
• Calm technical leadership across high-stakes cluster bring-ups and production reliability initiatives.

PREFERRED / BONUS EXPERIENCE
• Experience building P2P or RDMA-accelerated container/model-weight distribution systems across thousands of GPU nodes.
• Familiarity with both synchronous pre-training gang scheduling and elastic asynchronous RL actor-learner cluster workloads.
• Core contributions to Kubernetes WG-Serving / Kueue, Volcano, Ray on K8s, or Linux eBPF observability tools.

## Application Schema & Direct Apply (`apply_for_role`)

Candidates and authorized AI agents on behalf of a candidate can submit an application via the declarative WebMCP form tool `apply_for_role` on https://jobs.talent247.co/jobs/staff-ml-platform-cluster-reliability-engineer or via HTTP `POST https://jobs.talent247.co/api/public/jobs/staff-ml-platform-cluster-reliability-engineer/apply` (`application/json` or `multipart/form-data`). Non-browser HTTP agents must include header `X-Talent247-Intake-Token` obtained from `GET https://jobs.talent247.co/api/public/intake-token` (or automatically returned in `GET https://jobs.talent247.co/api/public/jobs/staff-ml-platform-cluster-reliability-engineer/form-schema`).

### Form Fields

- `full_name` (text, required): Full name
- `email` (email, required): Email
- `phone` (phone, optional): Phone
- `location` (text, optional): Location
- `linkedin_url` (url, optional): LinkedIn URL
- `cover_letter` (textarea, optional): Cover letter
- `consent` (checkbox, required): I consent to my data being processed for this application
- `cv_file` / `cv_base64` + `cv_filename`: Upload a CV in .pdf or .docx format (<= 10 MB) or provide an `https://` LinkedIn profile URL in `linkedin_url`.

### Alternative: Talent Pool Intake (`join_talent_pool`)

If this mandate is not an exact match, register once in the Talent247 Talent Pool at https://jobs.talent247.co/talent-pool (`https://jobs.talent247.co/talent-pool.md`) via WebMCP tool `join_talent_pool` or `POST https://jobs.talent247.co/api/public/talent-pool`.
