---
title: "Senior Distributed Data & Vector Indexing Engineer"
slug: "senior-distributed-data-vector-indexing-engineer"
id: "da3a54c8-28d8-4152-ac50-beb80cae9e4c"
department: "MLOps & Platform"
department_code: "OPS"
location: "New York, NY"
workplace_type: "Hybrid"
employment_type: "Full-time"
compensation: "$215k – $305k (annual package)"
is_remote: false
is_featured: false
headcount: 1
canonical_url: "https://jobs.talent247.co/jobs/senior-distributed-data-vector-indexing-engineer"
markdown_url: "https://jobs.talent247.co/jobs/senior-distributed-data-vector-indexing-engineer.md"
apply_endpoint: "https://jobs.talent247.co/api/public/jobs/senior-distributed-data-vector-indexing-engineer/apply"
schema_endpoint: "https://jobs.talent247.co/api/public/jobs/senior-distributed-data-vector-indexing-engineer/form-schema"
webmcp_tool: "apply_for_role"
updated_at: "2026-11-15T17:43:25.785Z"
content_signal: "search=yes, ai-input=yes, ai-train=no"
---

# Senior Distributed Data & Vector Indexing Engineer

## Position Telemetry

- **Reference:** `SENIOR-DISTRIBUTED`
- **Practice / Sector:** MLOps & Platform (`OPS`)
- **Location:** New York, NY
- **Workplace Type:** Hybrid
- **Employment Type:** Full-time
- **Compensation:** $215k – $305k (annual package)
- **Canonical HTML URL:** https://jobs.talent247.co/jobs/senior-distributed-data-vector-indexing-engineer

## Role Dossier & Specification

THE CLIENT & MANDATE
Our client is a premier real-time AI search and knowledge discovery platform headquartered in San Francisco with a fast-growing systems engineering hub in New York City. Answering breaking-news, market-moving financial, and technical queries accurately requires ingesting, rendering, deduplicating, embedding, and indexing millions of newly published or updated web pages every hour—with end-to-end crawl-to-searchable freshness measured in seconds rather than days.

They have engaged Talent247 to appoint a Senior Distributed Data & Vector Indexing Engineer in New York, NY. Joining their Search Infrastructure & Freshness Indexing team, the successful candidate will architect petabyte-scale streaming pipelines and hybrid lexical-vector indexers in Rust, Go, and C++.

KEY RESPONSIBILITIES
• Architect ultra-high-throughput streaming ingestion and content-extraction pipelines in Rust and Go processing hundreds of thousands of web documents, feeds, and structured datasets per second.
• Build real-time incremental vector and inverted-index storage tiers supporting sub-minute document visibility, instant tombstone deletions, and lock-free concurrent reads at tens of thousands of QPS.
• Optimize batch-and-streaming GPU embedding inference pipelines (dynamic micro-batching, INT8/binary vector quantization, delta-chunk re-embedding) minimizing GPU cost across hundreds of billions of passages.
• Design petabyte-scale content deduplication (SimHash/MinHash/URL canonicalization), spam/SEO-farm pre-filtering, and crawl-priority frontier schedulers reactive to live query spikes.
• Engineer tiered RAM / PCIe Gen5 NVMe / object-store index layouts (DiskANN / HNSW / custom LSM-inverted structures) balancing sub-30ms tail latency with aggressive infrastructure efficiency.
• Instrument end-to-end freshness SLAs, partition rebalancing automation, and multi-region active-active index replication.

TECHNICAL ENVIRONMENT & OFFERING
Stack: Rust, Go, C++20, Kafka / Redpanda, Ray Data, Apache Arrow / Parquet / Lance, custom HNSW/DiskANN/BM25 storage engines, RocksDB, ClickHouse, Kubernetes, bare-metal NVMe & GPU clusters. Workplace: Hybrid in New York, NY. Compensation: $215,000 – $305,000 base salary plus high-upside unicorn equity and full US benefits.

## Core Requirements

ESSENTIAL QUALIFICATIONS
• 5+ years of distributed systems, search-index infrastructure, or high-throughput streaming data platform engineering experience in Rust, C++, or Go.
• Deep systems expertise in storage engines, LSM-trees, inverted indices, or approximate nearest-neighbor vector indices (HNSW, IVF-PQ, Vamana/DiskANN) operating across terabytes of NVMe/RAM.
• Proven track record building low-latency streaming pipelines (Kafka/Redpanda/Flink/custom async Rust) sustaining hundreds of megabytes to gigabytes per second of sustained throughput.
• Strong Linux performance engineering skills across memory-mapped IO (`mmap`), `io_uring`, SIMD acceleration, network serialization, and tail-latency debugging.
• High ownership mindset thriving in lean systems squads operating planet-scale data infrastructure.

PREFERRED / BONUS EXPERIENCE
• Prior experience building web crawlers, real-time news/financial search indices, or billion-scale vector databases (Qdrant, Milvus, Vespa, Lucene/Tantivy internals).
• Experience orchestrating high-throughput GPU embedding/classification fleets alongside CPU streaming pipelines.
• Open-source contributions to Rust search/storage ecosystems (`tantivy`, `lancedb`, `qdrant`, `rocksdb`, `arrow-rs`).

## Application Schema & Direct Apply (`apply_for_role`)

Candidates and authorized AI agents on behalf of a candidate can submit an application via the declarative WebMCP form tool `apply_for_role` on https://jobs.talent247.co/jobs/senior-distributed-data-vector-indexing-engineer or via HTTP `POST https://jobs.talent247.co/api/public/jobs/senior-distributed-data-vector-indexing-engineer/apply` (`application/json` or `multipart/form-data`). Non-browser HTTP agents must include header `X-Talent247-Intake-Token` obtained from `GET https://jobs.talent247.co/api/public/intake-token` (or automatically returned in `GET https://jobs.talent247.co/api/public/jobs/senior-distributed-data-vector-indexing-engineer/form-schema`).

### Form Fields

- `full_name` (text, required): Full name
- `email` (email, required): Email
- `phone` (phone, optional): Phone
- `location` (text, optional): Location
- `linkedin_url` (url, optional): LinkedIn URL
- `cover_letter` (textarea, optional): Cover letter
- `consent` (checkbox, required): I consent to my data being processed for this application
- `cv_file` / `cv_base64` + `cv_filename`: Upload a CV in .pdf or .docx format (<= 10 MB) or provide an `https://` LinkedIn profile URL in `linkedin_url`.

### Alternative: Talent Pool Intake (`join_talent_pool`)

If this mandate is not an exact match, register once in the Talent247 Talent Pool at https://jobs.talent247.co/talent-pool (`https://jobs.talent247.co/talent-pool.md`) via WebMCP tool `join_talent_pool` or `POST https://jobs.talent247.co/api/public/talent-pool`.
