Skip to main content
TALENT247 // SPECIALIST AI EXECUTIVE SEARCH & TECHNICAL RECRUITMENT
OPS · MLOps & Platform

Senior Distributed Data & Vector Indexing Engineer

New York, NY · Full-time · Hybrid

AI Specialism

MLOps & Platform

Workplace

Hybrid

Location

New York, NY

Compensation

$215k – $305k

annual package

Role Dossier & Specification

REF: SENIOR-DISTRIBUTED

THE CLIENT & MANDATE

Our client is a premier real-time AI search and knowledge discovery platform headquartered in San Francisco with a fast-growing systems engineering hub in New York City. Answering breaking-news, market-moving financial, and technical queries accurately requires ingesting, rendering, deduplicating, embedding, and indexing millions of newly published or updated web pages every hour—with end-to-end crawl-to-searchable freshness measured in seconds rather than days.

They have engaged Talent247 to appoint a Senior Distributed Data & Vector Indexing Engineer in New York, NY. Joining their Search Infrastructure & Freshness Indexing team, the successful candidate will architect petabyte-scale streaming pipelines and hybrid lexical-vector indexers in Rust, Go, and C++.

KEY RESPONSIBILITIES

  • Architect ultra-high-throughput streaming ingestion and content-extraction pipelines in Rust and Go processing hundreds of thousands of web documents, feeds, and structured datasets per second.
  • Build real-time incremental vector and inverted-index storage tiers supporting sub-minute document visibility, instant tombstone deletions, and lock-free concurrent reads at tens of thousands of QPS.
  • Optimize batch-and-streaming GPU embedding inference pipelines (dynamic micro-batching, INT8/binary vector quantization, delta-chunk re-embedding) minimizing GPU cost across hundreds of billions of passages.
  • Design petabyte-scale content deduplication (SimHash/MinHash/URL canonicalization), spam/SEO-farm pre-filtering, and crawl-priority frontier schedulers reactive to live query spikes.
  • Engineer tiered RAM / PCIe Gen5 NVMe / object-store index layouts (DiskANN / HNSW / custom LSM-inverted structures) balancing sub-30ms tail latency with aggressive infrastructure efficiency.
  • Instrument end-to-end freshness SLAs, partition rebalancing automation, and multi-region active-active index replication.

TECHNICAL ENVIRONMENT & OFFERING

Stack: Rust, Go, C++20, Kafka / Redpanda, Ray Data, Apache Arrow / Parquet / Lance, custom HNSW/DiskANN/BM25 storage engines, RocksDB, ClickHouse, Kubernetes, bare-metal NVMe & GPU clusters. Workplace: Hybrid in New York, NY. Compensation: $215,000 – $305,000 base salary plus high-upside unicorn equity and full US benefits.

Core Requirements

ESSENTIAL QUALIFICATIONS

  • 5+ years of distributed systems, search-index infrastructure, or high-throughput streaming data platform engineering experience in Rust, C++, or Go.
  • Deep systems expertise in storage engines, LSM-trees, inverted indices, or approximate nearest-neighbor vector indices (HNSW, IVF-PQ, Vamana/DiskANN) operating across terabytes of NVMe/RAM.
  • Proven track record building low-latency streaming pipelines (Kafka/Redpanda/Flink/custom async Rust) sustaining hundreds of megabytes to gigabytes per second of sustained throughput.
  • Strong Linux performance engineering skills across memory-mapped IO (mmap), io_uring, SIMD acceleration, network serialization, and tail-latency debugging.
  • High ownership mindset thriving in lean systems squads operating planet-scale data infrastructure.

PREFERRED / BONUS EXPERIENCE

  • Prior experience building web crawlers, real-time news/financial search indices, or billion-scale vector databases (Qdrant, Milvus, Vespa, Lucene/Tantivy internals).
  • Experience orchestrating high-throughput GPU embedding/classification fleets alongside CPU streaming pipelines.
  • Open-source contributions to Rust search/storage ecosystems (tantivy, lancedb, qdrant, rocksdb, arrow-rs).

Position Telemetry

Location New York, NY (Hybrid)
Department MLOps & Platform
Employment type Full-time
Workplace type Hybrid
Salary range
$215k – $305k annual package