Skip to main content
TALENT247 // SPECIALIST AI EXECUTIVE SEARCH & TECHNICAL RECRUITMENT
INF · AI Infrastructure

Distributed Storage Engineer, Model Weight Streaming

New York, NY · Full-time · On-site

AI Specialism

AI Infrastructure

Workplace

On-site

Location

New York, NY

Compensation

$225k – $320k

annual package

Role Dossier & Specification

REF: DISTRIBUTED-STORAG

THE CLIENT & MANDATE

Our client is a premier New York City serverless GPU and AI compute platform powering instant-scaling inference, batch multimedia processing, and massive reinforcement learning sandboxes. When a customer scales a 400-gigabyte Mixture-of-Experts model from zero to 500 GPU replicas in three seconds, conventional container registries and POSIX network filesystems collapse—making their custom Rust distributed storage engine a foundational competitive moat.

They have partnered with Talent247 to hire a Distributed Storage Engineer for Model Weight Streaming onsite in New York, NY. The successful candidate will co-architect their content-addressable filesystem, tiered NVMe-oF caches, and zero-copy RDMA/PCIe weight-streaming protocols saturating hundreds of gigabits per second per GPU node.

KEY RESPONSIBILITIES

  • Design and scale a custom Rust distributed, content-addressable storage and volume engine serving petabytes of model weights, container layers, and training checkpoints across tens of thousands of concurrent serverless workers.
  • Engineer ultra-high-throughput tiered caching hierarchies spanning local PCIe Gen5 NVMe arrays, peer-to-peer cluster caches, erasure-coded regional storage nodes, and cloud object stores.
  • Implement zero-copy io_uring, direct IO (O_DIRECT), and GPUDirect Storage (cuFile / DMA) streaming paths loading safetensors shards from network/NVMe directly into GPU HBM at line rate.
  • Build transactional copy-on-write (CoW) volume snapshotting and instant branching primitives enabling million-container RL coding/data sandboxes to fork terabyte datasets in milliseconds.
  • Design consensus, metadata sharding, background garbage collection, and bit-rot scrubbing protocols upholding strict durability and consistency guarantees under node churn.
  • Benchmark and model tail-latency behavior under extreme thundering-herd fan-out workloads using deterministic simulation testing.

TECHNICAL ENVIRONMENT & OFFERING

Stack: Rust, io_uring, FUSE / NFS / custom RPC protocols, NVMe / NVMe-oF, RDMA, erasure coding (Reed-Solomon), Raft/Paxos metadata stores, S3/GCS backends, Linux bare-metal & cloud GPU fleets. Workplace: Onsite in New York, NY. Compensation: $225,000 – $320,000 base salary plus substantial equity ownership, full healthcare, daily meals, and relocation support.

Core Requirements

ESSENTIAL QUALIFICATIONS

  • 5+ years of distributed storage, filesystem, database storage-engine, or high-performance networked IO engineering experience in Rust, C++, or C.
  • Deep understanding of Linux storage stacks (io_uring, NVMe block layer, page cache vs. O_DIRECT, FUSE, filesystem metadata layouts) and networked storage throughput tuning.
  • Strong distributed systems foundations across consistent hashing, replication, erasure coding, lease/lock management, and crash recovery.
  • Proven ability to reason quantitatively about disk IOPS, network bisection bandwidth, read/write amplification, and thundering-herd cache stampedes.
  • Passion for writing clean, deterministic, simulation-tested Rust systems code.

PREFERRED / BONUS EXPERIENCE

  • Experience with GPUDirect Storage, RDMA/RoCEv2 data transfers, or P2P content-addressed distribution of massive ML artifacts (safetensors, GGUF, OCI layers).
  • Prior work on distributed filesystems or object/block stores (Ceph, Lustre, 3FS/Tectonic/JuiceFS-class architectures, FoundationDB, RocksDB).
  • Experience with deterministic simulation testing (FoundationDB / turmoil-style fault injection) in Rust.

Position Telemetry

Location New York, NY
Department AI Infrastructure
Employment type Full-time
Workplace type On-site
Salary range
$225k – $320k annual package