Skip to main content
TALENT247 // SPECIALIST AI EXECUTIVE SEARCH & TECHNICAL RECRUITMENT
OPS · MLOps & Platform

Senior ML Data & Preference Pipeline Engineer

San Francisco, CA · Full-time · On-site

AI Specialism

MLOps & Platform

Workplace

On-site

Location

San Francisco, CA

Compensation

$205k – $300k

annual package

Role Dossier & Specification

REF: SENIOR-ML-DATA-PRE

THE CLIENT & MANDATE

Our client is a category-defining AI-native software engineering company in San Francisco whose coding models and autonomous agents process billions of code completions and multi-file agentic trajectories every week. Turning live developer interaction patterns—strictly from opted-in telemetry and synthetic sandbox runs while guaranteeing airtight zero-data-retention isolation for enterprise tenants—into clean, real-time post-training preference signals is a cornerstone of their data flywheel.

They have engaged Talent247 to hire a Senior ML Data & Preference Pipeline Engineer onsite in San Francisco, CA. Working shoulder-to-shoulder with code-model researchers, the appointed engineer will build ultra-high-throughput streaming ingestion, privacy-enforcing sanitization, and automated preference-pair synthesis engines feeding daily SFT and RL runs.

KEY RESPONSIBILITIES

  • Architect high-throughput Rust, Go, and Python streaming pipelines ingesting tens of gigabytes per second of structured editor/agent telemetry into ClickHouse, Kafka, and petabyte columnar lakes.
  • Build deterministic, provably audited privacy and tenant-isolation gates ensuring enterprise zero-data-retention traffic, secrets/API keys, and PII never enter persistence or training pipelines.
  • Design counterfactual preference-extraction pipelines reconstructing exact repository states, accepted vs. modified multi-line diffs, compiler/linter outcomes, and long-horizon survival of agent-authored code.
  • Engineer fast AST-aware deduplication, near-duplicate MinHash clustering, syntax/test-validity filtering, and dynamic curriculum-mixing pipelines feeding multi-node PyTorch training clusters.
  • Construct real-time online experiment attribution dashboards linking training-mixture ablations directly to sub-hour live user acceptance, edit-survival, and latency metrics.
  • Optimize petabyte storage layouts (Lance / Parquet / Arrow) and zero-copy tokenized shard streaming to keep thousand-GPU training runs 100% compute-bound.

TECHNICAL ENVIRONMENT & OFFERING

Stack: Rust, Python, ClickHouse, Kafka / Redpanda, Ray Data, Lance / Apache Arrow / Parquet, tree-sitter, PyTorch data loaders, Kubernetes, AWS/GCP. Workplace: Onsite in San Francisco, CA. Compensation: $205,000 – $300,000 base salary plus top-tier hyper-growth equity, comprehensive benefits, and daily meals.

Core Requirements

ESSENTIAL QUALIFICATIONS

  • 5+ years of data infrastructure, ML data platform, or high-throughput streaming systems engineering experience operating at multi-terabyte-per-day scale.
  • Strong systems programming fluency in Rust, C++, or Go paired with idiomatic Python for ML dataset tooling and PyTorch integration.
  • Production mastery of analytical/streaming stores such as ClickHouse, Kafka/Redpanda, Ray Data, and modern columnar formats (Parquet, Arrow, Lance).
  • Proven experience building ML training data engines—covering deduplication, quality filtering, secret/PII scrubbing, or RLHF preference-pair construction.
  • Extreme rigor around data correctness, privacy boundary enforcement, and reproducible dataset lineage.

PREFERRED / BONUS EXPERIENCE

  • Experience parsing and transforming source-code corpora at scale using tree-sitter, compiler ASTs, or git commit-graph mining.
  • Background building online feature/bandit feedback loops or real-time A/B experimentation platforms.
  • Familiarity with LLM tokenization internals, sequence packing, and deterministic multi-epoch data mixing.

Position Telemetry

Location San Francisco, CA
Department MLOps & Platform
Employment type Full-time
Workplace type On-site
Salary range
$205k – $300k annual package