---
title: "Senior ML Data & Preference Pipeline Engineer"
slug: "senior-ml-data-preference-pipeline-engineer"
id: "a7b2a7fd-943d-48a2-bc1a-ccbc4fac5296"
department: "MLOps & Platform"
department_code: "OPS"
location: "San Francisco, CA"
workplace_type: "On-site"
employment_type: "Full-time"
compensation: "$205k – $300k (annual package)"
is_remote: false
is_featured: false
headcount: 1
canonical_url: "https://jobs.talent247.co/jobs/senior-ml-data-preference-pipeline-engineer"
markdown_url: "https://jobs.talent247.co/jobs/senior-ml-data-preference-pipeline-engineer.md"
apply_endpoint: "https://jobs.talent247.co/api/public/jobs/senior-ml-data-preference-pipeline-engineer/apply"
schema_endpoint: "https://jobs.talent247.co/api/public/jobs/senior-ml-data-preference-pipeline-engineer/form-schema"
webmcp_tool: "apply_for_role"
updated_at: "2026-11-15T17:39:04.630Z"
content_signal: "search=yes, ai-input=yes, ai-train=no"
---

# Senior ML Data & Preference Pipeline Engineer

## Position Telemetry

- **Reference:** `SENIOR-ML-DATA-PRE`
- **Practice / Sector:** MLOps & Platform (`OPS`)
- **Location:** San Francisco, CA
- **Workplace Type:** On-site
- **Employment Type:** Full-time
- **Compensation:** $205k – $300k (annual package)
- **Canonical HTML URL:** https://jobs.talent247.co/jobs/senior-ml-data-preference-pipeline-engineer

## Role Dossier & Specification

THE CLIENT & MANDATE
Our client is a category-defining AI-native software engineering company in San Francisco whose coding models and autonomous agents process billions of code completions and multi-file agentic trajectories every week. Turning live developer interaction patterns—strictly from opted-in telemetry and synthetic sandbox runs while guaranteeing airtight zero-data-retention isolation for enterprise tenants—into clean, real-time post-training preference signals is a cornerstone of their data flywheel.

They have engaged Talent247 to hire a Senior ML Data & Preference Pipeline Engineer onsite in San Francisco, CA. Working shoulder-to-shoulder with code-model researchers, the appointed engineer will build ultra-high-throughput streaming ingestion, privacy-enforcing sanitization, and automated preference-pair synthesis engines feeding daily SFT and RL runs.

KEY RESPONSIBILITIES
• Architect high-throughput Rust, Go, and Python streaming pipelines ingesting tens of gigabytes per second of structured editor/agent telemetry into ClickHouse, Kafka, and petabyte columnar lakes.
• Build deterministic, provably audited privacy and tenant-isolation gates ensuring enterprise zero-data-retention traffic, secrets/API keys, and PII never enter persistence or training pipelines.
• Design counterfactual preference-extraction pipelines reconstructing exact repository states, accepted vs. modified multi-line diffs, compiler/linter outcomes, and long-horizon survival of agent-authored code.
• Engineer fast AST-aware deduplication, near-duplicate MinHash clustering, syntax/test-validity filtering, and dynamic curriculum-mixing pipelines feeding multi-node PyTorch training clusters.
• Construct real-time online experiment attribution dashboards linking training-mixture ablations directly to sub-hour live user acceptance, edit-survival, and latency metrics.
• Optimize petabyte storage layouts (Lance / Parquet / Arrow) and zero-copy tokenized shard streaming to keep thousand-GPU training runs 100% compute-bound.

TECHNICAL ENVIRONMENT & OFFERING
Stack: Rust, Python, ClickHouse, Kafka / Redpanda, Ray Data, Lance / Apache Arrow / Parquet, tree-sitter, PyTorch data loaders, Kubernetes, AWS/GCP. Workplace: Onsite in San Francisco, CA. Compensation: $205,000 – $300,000 base salary plus top-tier hyper-growth equity, comprehensive benefits, and daily meals.

## Core Requirements

ESSENTIAL QUALIFICATIONS
• 5+ years of data infrastructure, ML data platform, or high-throughput streaming systems engineering experience operating at multi-terabyte-per-day scale.
• Strong systems programming fluency in Rust, C++, or Go paired with idiomatic Python for ML dataset tooling and PyTorch integration.
• Production mastery of analytical/streaming stores such as ClickHouse, Kafka/Redpanda, Ray Data, and modern columnar formats (Parquet, Arrow, Lance).
• Proven experience building ML training data engines—covering deduplication, quality filtering, secret/PII scrubbing, or RLHF preference-pair construction.
• Extreme rigor around data correctness, privacy boundary enforcement, and reproducible dataset lineage.

PREFERRED / BONUS EXPERIENCE
• Experience parsing and transforming source-code corpora at scale using `tree-sitter`, compiler ASTs, or git commit-graph mining.
• Background building online feature/bandit feedback loops or real-time A/B experimentation platforms.
• Familiarity with LLM tokenization internals, sequence packing, and deterministic multi-epoch data mixing.

## Application Schema & Direct Apply (`apply_for_role`)

Candidates and authorized AI agents on behalf of a candidate can submit an application via the declarative WebMCP form tool `apply_for_role` on https://jobs.talent247.co/jobs/senior-ml-data-preference-pipeline-engineer or via HTTP `POST https://jobs.talent247.co/api/public/jobs/senior-ml-data-preference-pipeline-engineer/apply` (`application/json` or `multipart/form-data`). Non-browser HTTP agents must include header `X-Talent247-Intake-Token` obtained from `GET https://jobs.talent247.co/api/public/intake-token` (or automatically returned in `GET https://jobs.talent247.co/api/public/jobs/senior-ml-data-preference-pipeline-engineer/form-schema`).

### Form Fields

- `full_name` (text, required): Full name
- `email` (email, required): Email
- `phone` (phone, optional): Phone
- `location` (text, optional): Location
- `linkedin_url` (url, optional): LinkedIn URL
- `cover_letter` (textarea, optional): Cover letter
- `consent` (checkbox, required): I consent to my data being processed for this application
- `cv_file` / `cv_base64` + `cv_filename`: Upload a CV in .pdf or .docx format (<= 10 MB) or provide an `https://` LinkedIn profile URL in `linkedin_url`.

### Alternative: Talent Pool Intake (`join_talent_pool`)

If this mandate is not an exact match, register once in the Talent247 Talent Pool at https://jobs.talent247.co/talent-pool (`https://jobs.talent247.co/talent-pool.md`) via WebMCP tool `join_talent_pool` or `POST https://jobs.talent247.co/api/public/talent-pool`.
