Skip to main content
TALENT247 // SPECIALIST AI EXECUTIVE SEARCH & TECHNICAL RECRUITMENT
INF · AI Infrastructure

Senior Systems Engineer, Ultra-Low-Latency Inference Cloud

Austin, TX · Full-time · Hybrid

AI Specialism

AI Infrastructure

Workplace

Hybrid

Location

Austin, TX

Compensation

$185k – $255k

annual package

Role Dossier & Specification

REF: SENIOR-SYSTEMS-ENG

THE CLIENT & MANDATE

Our client is a breakthrough custom AI silicon and inference-cloud provider operating massive dedicated AI supercomputing clusters across North America, with a flagship systems engineering hub in Austin, TX. Because their custom accelerators generate LLM tokens at 2,000+ tokens per second per stream—an order of magnitude faster than conventional GPU clouds—standard HTTP gateways and Python serving wrappers immediately become the bottleneck.

They have engaged Talent247 to hire a Senior Systems Engineer for their Ultra-Low-Latency Inference Cloud in Austin, TX. The appointed engineer will build ultra-lean Rust/C++/Go streaming data planes, stateful KV-cache-aware request routers, and multi-region failover fabrics capable of sustaining millions of tokens per second with single-digit millisecond gateway overhead.

KEY RESPONSIBILITIES

  • Architect high-concurrency, zero-copy Rust and Go token-streaming gateways (HTTP/2, HTTP/3, gRPC, SSE, WebSockets) adding <2ms p99 overhead on 2,000+ tok/s reasoning and coding streams.
  • Design distributed prefix-cache-aware and session-affine request schedulers that maximize on-silicon KV-cache hit rates across multi-turn agentic coding and tool-calling workloads.
  • Build continuous-batching admission controllers, multi-tier token-bucket rate limiters, and fairness queues balancing interactive low-latency API traffic against massive asynchronous batch/RL rollout jobs.
  • Engineer zero-downtime weight-reload orchestration and fast cross-datacenter traffic draining over bare-metal Kubernetes clusters connected directly to custom AI accelerators.
  • Instrument nanosecond-to-millisecond distributed tracing (OpenTelemetry, eBPF) tracking time-to-first-token (TTFT), inter-token latency (ITL), and queue wait across every hop.
  • Partner closely with compiler and runtime engineers to expose structured outputs, speculative decoding, and reasoning-effort controls cleanly via industry-standard streaming chat-completion APIs.

TECHNICAL ENVIRONMENT & OFFERING

Stack: Rust, Go, C++20, Tokio/io_uring, Envoy, gRPC, Kubernetes (bare-metal), Redis/Dragonfly, ClickHouse, Prometheus/eBPF, Linux high-speed networking. Workplace: Hybrid in Austin, TX. Compensation: $185,000 – $255,000 base salary plus meaningful equity, relocation assistance, and comprehensive US benefits.

Core Requirements

ESSENTIAL QUALIFICATIONS

  • 5+ years of systems, networking, or cloud data-plane engineering experience building ultra-low-latency, high-throughput distributed services in Rust, C++, or Go.
  • Deep understanding of Linux networking internals, async event loops (Tokio/epoll/io_uring), HTTP/2 & gRPC framing, backpressure handling, and tail-latency elimination.
  • Production experience operating stateful or hardware-attached services on Kubernetes, custom schedulers, or Envoy/service-mesh control planes.
  • Familiarity with LLM inference serving mechanics—including continuous batching, prefill vs. decode disaggregation, KV-cache locality, and streaming tokenization.
  • Strong operational discipline around load testing, flame-graph profiling, and multi-region high availability.

PREFERRED / BONUS EXPERIENCE

  • Direct contributions to LLM serving routers or gateways (vLLM router, SGLang router, Dynamo, Envoy AI Gateway, Pingora).
  • Experience with RDMA/RoCEv2 networking, DPDK/XDP kernel-bypass, or custom hardware host-interface drivers.
  • Track record scaling developer-facing commercial API clouds under strict 99.99% uptime SLAs.

Position Telemetry

Location Austin, TX (Hybrid)
Department AI Infrastructure
Employment type Full-time
Workplace type Hybrid
Salary range
$185k – $255k annual package