Skip to main content
TALENT247 // SPECIALIST AI EXECUTIVE SEARCH & TECHNICAL RECRUITMENT
INF · AI Infrastructure

Senior GPU Cluster & RDMA Networking Engineer

San Francisco, CA · Full-time · Hybrid

AI Specialism

AI Infrastructure

Workplace

Hybrid

Location

San Francisco, CA

Compensation

$225k – $315k

annual package

Role Dossier & Specification

REF: SENIOR-GPU-CLUSTER

THE CLIENT & MANDATE

Our client is a multi-billion-dollar AI Acceleration Cloud headquartered in San Francisco operating massive bare-metal clusters of NVIDIA H100, H200, B200, and GB200 NVL72 superpods across North America and Europe. Delivering hyperscale-beating distributed training and disaggregated MoE inference performance depends on lossless, microsecond-tuned 400G/800G InfiniBand and RoCEv2 RDMA fabrics engineered down to the NIC register and NCCL transport layer.

They have engaged Talent247 to hire a Senior GPU Cluster & RDMA Networking Engineer in San Francisco, CA. Joining their Bare-Metal Supercomputing Infrastructure group, the successful candidate will architect rail-optimized AI fabrics, tune NCCL/NVSHMEM collective transports, and build real-time RDMA telemetry across tens of thousands of GPUs.

KEY RESPONSIBILITIES

  • Architect, deploy, and scale non-blocking rail-optimized 400G and 800G NVIDIA Quantum-2/X800 InfiniBand and Spectrum-X / Broadcom Tomahawk5 RoCEv2 fabrics for multi-thousand-GPU clusters.
  • Co-optimize NCCL, NVSHMEM, and libfabric/UCX collective communication parameters—tuning congestion control (DCQCN, ECN, PFC, adaptive routing), QP scaling, and PCIe/NVSwitch topology mappings.
  • Build automated pre-flight fabric qualification suites (ib_write_bw, nccl-tests, bisectional all-reduce/all-to-all sweeps) and continuous link-flap/BER optical degradation detectors that quarantine degraded rails before training jobs stall.
  • Engineer secure multi-tenant RDMA isolation (InfiniBand PKeys, EVPN-VXLAN / SRv6 + RoCEv2 ACLs, ConnectX-7/8 BlueField-3 DPU offloads) for dedicated enterprise training pods.
  • Partner directly with inference and training kernel engineers to optimize low-latency internode expert-parallel all-to-all dispatch and KV-cache RDMA transfers.
  • Drive vendor silicon/firmware root-cause debugging across switches, optical transceivers, DPUs, and Linux RDMA (rdma-core / MLNX_OFED / DOCA) kernel stacks.

TECHNICAL ENVIRONMENT & OFFERING

Stack: 400G/800G InfiniBand (UFM) & RoCEv2 (SONiC / EOS / Cumulus), ConnectX-7/8 & BlueField-3 DPUs, GPUDirect RDMA, NCCL/NVSHMEM, Linux networking, Python/Go/Rust automation, Prometheus/eBPF. Workplace: Hybrid in San Francisco, CA. Compensation: $225,000 – $315,000 base salary plus substantial equity ownership and full US benefits.

Core Requirements

ESSENTIAL QUALIFICATIONS

  • 5+ years of high-performance datacenter networking, HPC interconnect, or AI GPU fabric engineering experience.
  • Deep hands-on mastery of InfiniBand and/or lossless Ethernet RoCEv2 at 200G/400G/800G scale—including BGP CLOS fabrics, PFC/ECN/DCQCN congestion control, adaptive routing, and optical link physics.
  • Strong command of GPUDirect RDMA, PCIe root-complex / ACS / IOMMU interactions, rdma-core / DOCA / OFED drivers, and nccl-tests performance profiling.
  • Proficient infrastructure software skills in Python, Go, or Rust for switch/fabric automation, zero-touch provisioning, and streaming telemetry.
  • Proven ability to isolate microsecond tail-latency anomalies across multi-rack synchronous AI workloads.

PREFERRED / BONUS EXPERIENCE

  • Direct experience bringing up NVIDIA GB200 NVL72 clusters, Spectrum-X Ethernet fabrics, or multi-tenant BlueField-3 DPU virtualization.
  • Familiarity with custom NCCL net-plugins, NVSHMEM internode MoE dispatch, or Ultra Ethernet Consortium (UEC) transports.
  • Prior tenure building AI superclusters at a hyperscaler, neocloud, frontier AI lab, or national supercomputing facility.

Position Telemetry

Location San Francisco, CA (Hybrid)
Department AI Infrastructure
Employment type Full-time
Workplace type Hybrid
Salary range
$225k – $315k annual package