Skip to main content
TALENT247 // SPECIALIST AI EXECUTIVE SEARCH & TECHNICAL RECRUITMENT
INF · AI Infrastructure

Senior LLM Inference Engine Engineer (MoE & LoRA)

Redwood City, CA · Full-time · Hybrid

AI Specialism

AI Infrastructure

Workplace

Hybrid

Location

Redwood City, CA

Compensation

$225k – $325k

annual package

Role Dossier & Specification

REF: SENIOR-LLM-INFEREN

THE CLIENT & MANDATE

Our client is a multi-billion-dollar generative AI inference and post-training cloud headquartered in Redwood City, CA, founded by core creators of modern deep-learning frameworks. Serving trillions of daily tokens for top AI-native products and enterprises, their inference engine powers ultra-fast open-weight Mixture-of-Experts (MoE) models alongside thousands of concurrently hot-swapped custom LoRA and RL-tuned adapters on shared GPU pods.

They have retained Talent247 to appoint a Senior LLM Inference Engine Engineer (MoE & LoRA) in Redwood City, CA. Working at the core of their proprietary C++/CUDA/Python serving runtime, the successful candidate will push distributed MoE expert parallelism, disaggregated KV-caching, and multi-LoRA batching to industry-leading throughput-per-dollar frontiers.

KEY RESPONSIBILITIES

  • Architect and optimize distributed Mixture-of-Experts (MoE) serving topologies—spanning wide expert parallelism (EP), RDMA/NVLink low-latency all-to-all token dispatch (DeepEP-class), and dynamic expert load balancing across multi-node Hopper and Blackwell pods.
  • Engineer zero-overhead multi-LoRA batched GEMM kernels and tiered GPU/CPU/NVMe adapter caches supporting thousands of customer fine-tuned models on shared base-weight replicas without cold-start stalls.
  • Implement disaggregated prefill/decode scheduling, speculative multi-token draft-verify loops (MTP / EAGLE-style), and grammar-constrained structured decoding with near-zero CPU overhead.
  • Optimize FP8, NVFP4, and mixed-precision quantization schemes preserving full reasoning and tool-calling fidelity on 100B–600B+ parameter architectures.
  • Deliver day-zero optimized support for newly released open-weight LLM, VLM, and audio-text architectures within hours of weight publication.
  • Profile end-to-end CUDA Graphs, NCCL/NVSHMEM collectives, and Python/C++ scheduler overhead using Nsight Compute and Nsight Systems.

TECHNICAL ENVIRONMENT & OFFERING

Stack: C++20, CUDA, CUTLASS, Triton, Python, PyTorch, custom SGLang/vLLM/TensorRT-LLM-class inference runtime, NVSHMEM/RDMA, H100/H200/B200 GPU clusters. Workplace: Hybrid in Redwood City, CA. Compensation: $225,000 – $325,000 base salary plus high-growth unicorn equity and full US benefits.

Core Requirements

ESSENTIAL QUALIFICATIONS

  • 5+ years of deep learning runtime, GPU kernel, or LLM inference engine engineering experience in modern C++, CUDA, and Python.
  • Deep production familiarity with modern LLM serving internals (continuous batching, paged KV caches, chunked prefill, CUDA Graphs, tensor & expert parallelism).
  • Hands-on experience authoring or optimizing CUDA/CUTLASS/Triton kernels for attention, grouped GEMMs, MoE dispatch, or quantized matmuls (FP8/INT4/FP4).
  • Strong command of multi-GPU / multi-node interconnect performance (NVLink, InfiniBand/RoCE GPUDirect RDMA) and latency-throughput Pareto trade-offs.
  • Track record shipping rock-solid production runtime code handling billions of requests without memory leaks or numerical regressions.

PREFERRED / BONUS EXPERIENCE

  • Core contributions to SGLang, vLLM, TensorRT-LLM, FlashInfer, DeepEP, Punica/S-LoRA, or PyTorch compiler/runtime internals.
  • Experience implementing multi-token prediction (MTP), EAGLE speculative decoding, or grammar-guided FSM token masking (xgrammar/outlines).
  • Familiarity with AMD Instinct (ROCm/HIP) or NVIDIA Blackwell GB200 NVL72 rack-scale optimization.

Position Telemetry

Location Redwood City, CA (Hybrid)
Department AI Infrastructure
Employment type Full-time
Workplace type Hybrid
Headcount 2 openings
Salary range
$225k – $325k annual package