Skip to main content
TALENT247 // SPECIALIST AI EXECUTIVE SEARCH & TECHNICAL RECRUITMENT
OPS · MLOps & Platform

Senior GPU Control Plane & Kubernetes Engineer

Redwood City, CA · Full-time · Hybrid

AI Specialism

MLOps & Platform

Workplace

Hybrid

Location

Redwood City, CA

Compensation

$215k – $295k

annual package

Role Dossier & Specification

REF: SENIOR-GPU-CONTROL

THE CLIENT & MANDATE

Our client is a multi-billion-dollar AI inference and post-training cloud in Redwood City, CA operating tens of thousands of H100, H200, and B200 GPUs across global cloud and bare-metal regions. Orchestrating multi-node Mixture-of-Experts replicas, serverless burst autoscaling, and multi-hundred-gigabyte weight streaming with sub-second cold starts requires a purpose-built Kubernetes GPU control plane far beyond standard cloud autoscalers.

They have retained Talent247 to hire a Senior GPU Control Plane & Kubernetes Engineer in Redwood City, CA. Joining their Core Compute Infrastructure team, the successful candidate will architect custom Go and Rust Kubernetes operators, topology-aware multi-cluster schedulers, and P2P/RDMA model-weight distribution engines.

KEY RESPONSIBILITIES

  • Design and scale custom Kubernetes controllers, operators, and scheduler plugins in Go and Rust orchestrating multi-node GPU inference pods (NVLink/InfiniBand gang-scheduled replicas) across 20+ global regions.
  • Engineer ultra-fast GPU replica cold-start pipelines—combining pre-warmed container pools, P2P/RDMA peer-to-peer weight broadcasting, and direct NVMe-to-GPU (cuFile/GPUDirect Storage) weight loading—bringing 100B+ models online in seconds.
  • Build predictive, queue-depth- and KV-cache-pressure-aware autoscalers seamlessly bridging serverless multi-tenant GPU pools and dedicated enterprise reservations.
  • Implement zero-downtime rolling upgrades, live GPU health telemetry (DCGM/Xid/ECC/link-flap detection), and automated node replacement across heterogeneous cloud and neocloud providers.
  • Architect control-plane isolation, multi-tenant network policies (Cilium/eBPF), and high-throughput global Envoy/Rust ingress routing.
  • Drive cluster bin-packing efficiency and spot/reserved capacity brokering delivering double-digit millions in annual GPU infrastructure savings.

TECHNICAL ENVIRONMENT & OFFERING

Stack: Go, Rust, Kubernetes (client-go, controller-runtime, custom schedulers, DRA), Cilium/eBPF, Envoy, GPUDirect Storage / RDMA, Prometheus/VictoriaMetrics, AWS/GCP/OCI & bare-metal GPU clouds. Workplace: Hybrid in Redwood City, CA. Compensation: $215,000 – $295,000 base salary plus high-value unicorn equity and comprehensive US benefits.

Core Requirements

ESSENTIAL QUALIFICATIONS

  • 5+ years of cloud infrastructure, distributed control-plane, or Kubernetes platform engineering experience in Go or Rust.
  • Deep production mastery of Kubernetes internals: writing custom CRDs/controllers, custom scheduler plugins, pod lifecycle hooks, and CNI/CSI/device-plugin integrations at multi-thousand-node scale.
  • Hands-on experience operating NVIDIA GPU clusters in production (MIG/MPS, NVLink/InfiniBand topology awareness, DCGM telemetry, container toolkit internals).
  • Strong systems fundamentals in Linux networking, high-throughput artifact/image distribution, and distributed consensus (etcd/Raft).
  • Proven operational ownership of 99.99% uptime customer-facing cloud control planes.

PREFERRED / BONUS EXPERIENCE

  • Direct experience optimizing multi-hundred-gigabyte LLM weight loading or container image streaming (Dragonfly, Kraken, Stargz/SOCI, P2P RDMA caches).
  • Experience orchestrating multi-node disaggregated inference (prefill/decode pools) or gang-scheduled LeaderWorkerSet (LWS) workloads.
  • Upstream contributions to Kubernetes, LeaderWorkerSet, Kueue, Cilium, or CNCF projects.

Position Telemetry

Location Redwood City, CA (Hybrid)
Department MLOps & Platform
Employment type Full-time
Workplace type Hybrid
Salary range
$215k – $295k annual package