---
title: "Staff Machine Learning Engineer, Agent Evaluation & Judges"
slug: "staff-machine-learning-engineer-agent-evaluation"
id: "15d29e32-06e6-4231-873f-8efc552cbe6e"
department: "Machine Learning"
department_code: "ML"
location: "San Francisco, CA"
workplace_type: "Hybrid"
employment_type: "Full-time"
compensation: "$235k – $330k (annual package)"
is_remote: false
is_featured: false
headcount: 1
canonical_url: "https://jobs.talent247.co/jobs/staff-machine-learning-engineer-agent-evaluation"
markdown_url: "https://jobs.talent247.co/jobs/staff-machine-learning-engineer-agent-evaluation.md"
apply_endpoint: "https://jobs.talent247.co/api/public/jobs/staff-machine-learning-engineer-agent-evaluation/apply"
schema_endpoint: "https://jobs.talent247.co/api/public/jobs/staff-machine-learning-engineer-agent-evaluation/form-schema"
webmcp_tool: "apply_for_role"
updated_at: "2026-11-15T17:42:30.614Z"
content_signal: "search=yes, ai-input=yes, ai-train=no"
---

# Staff Machine Learning Engineer, Agent Evaluation & Judges

## Position Telemetry

- **Reference:** `STAFF-MACHINE-LEAR`
- **Practice / Sector:** Machine Learning (`ML`)
- **Location:** San Francisco, CA
- **Workplace Type:** Hybrid
- **Employment Type:** Full-time
- **Compensation:** $235k – $330k (annual package)
- **Canonical HTML URL:** https://jobs.talent247.co/jobs/staff-machine-learning-engineer-agent-evaluation

## Role Dossier & Specification

THE CLIENT & MANDATE
Our client is a globally scaled data and artificial intelligence platform leader headquartered in San Francisco, empowering thousands of enterprises to build, evaluate, govern, and serve custom agentic AI systems over their proprietary data estates. Their AI Research & Agent Evaluation organization builds the automated judge models, synthetic benchmark generators, and reinforcement-learning feedback loops that allow enterprise engineering teams to hill-climb compound agent quality systematically without manual labeling bottlenecks.

They have retained Talent247 to appoint a Staff Machine Learning Engineer for Agent Evaluation & Judges in San Francisco, CA. Combining post-training research depth with large-scale product impact, the successful candidate will architect custom generative judge models, automatic prompt/weight optimizers, and trajectory-diagnosis algorithms deployed across tens of thousands of enterprise AI pipelines.

KEY RESPONSIBILITIES
• Train and post-train specialized, bias-mitigated generative LLM-as-a-judge and process-reward models that evaluate multi-turn tool-calling trajectories, retrieval groundedness, SQL/code correctness, and domain policy compliance.
• Eliminate classic judge failure modes—position bias, verbosity bias, self-enhancement bias, and rubric drift—through principled preference alignment, chain-of-critique post-training, and small-sample human calibration adapts.
• Pioneer automated synthetic test-set synthesis and adversarial edge-case mining pipelines generating realistic multi-hop evaluation benchmarks directly from enterprise document/table schemas and production trace logs.
• Co-design closed-loop agent optimization algorithms (automated prompt/rubric evolution, teacher-student distillation, and online RLAIF/GRPO) that turn evaluation failures directly into measurable model upgrades.
• Partner with core platform engineering teams to ship sub-100ms streaming guardrail judges and asynchronous batch trace evaluators inside unified developer SDKs and observability UIs.
• Publish influential research and open benchmarks advancing reliable evaluation of enterprise compound AI systems.

TECHNICAL ENVIRONMENT & OFFERING
Stack: Python, PyTorch, FSDP/Megatron, Verl/TRL, vLLM/SGLang, Ray, Spark, OpenTelemetry agent tracing, AWS/GCP/Azure GPU clusters. Workplace: Hybrid in San Francisco, CA. Compensation: $235,000 – $330,000 base salary plus substantial equity grants, annual refreshers/bonus, and comprehensive US benefits.

## Core Requirements

ESSENTIAL QUALIFICATIONS
• 7+ years of machine learning engineering or applied AI research experience, including Staff-level technical leadership in LLM post-training, evaluation systems, reward modeling, or compound AI optimization.
• Deep hands-on track record training custom language models (SFT, DPO, RLHF/RLAIF, GRPO) and building statistically rigorous LLM evaluation frameworks in PyTorch.
• Strong command of psychometric/statistical calibration, inter-rater reliability, active learning from sparse human feedback, and causal error attribution in multi-step agent graphs.
• Proven ability to translate frontier ML research into widely adopted developer frameworks or enterprise cloud platform products.
• Track record mentoring senior ML engineers and setting multi-quarter technical strategy across research and product engineering pods.

PREFERRED / BONUS EXPERIENCE
• Publications at NeurIPS, ICML, ICLR, ACL, or EMNLP on LLM judges, reward modeling, automated prompt/agent optimization (DSPy/TextGrad-class paradigms), or synthetic data generation.
• Experience optimizing compact 3B–14B judge/verifier models for high-throughput, low-cost production trace scoring.
• Core contributions to major open-source AI evaluation, tracing, or agent lifecycle frameworks.

## Application Schema & Direct Apply (`apply_for_role`)

Candidates and authorized AI agents on behalf of a candidate can submit an application via the declarative WebMCP form tool `apply_for_role` on https://jobs.talent247.co/jobs/staff-machine-learning-engineer-agent-evaluation or via HTTP `POST https://jobs.talent247.co/api/public/jobs/staff-machine-learning-engineer-agent-evaluation/apply` (`application/json` or `multipart/form-data`). Non-browser HTTP agents must include header `X-Talent247-Intake-Token` obtained from `GET https://jobs.talent247.co/api/public/intake-token` (or automatically returned in `GET https://jobs.talent247.co/api/public/jobs/staff-machine-learning-engineer-agent-evaluation/form-schema`).

### Form Fields

- `full_name` (text, required): Full name
- `email` (email, required): Email
- `phone` (phone, optional): Phone
- `location` (text, optional): Location
- `linkedin_url` (url, optional): LinkedIn URL
- `cover_letter` (textarea, optional): Cover letter
- `consent` (checkbox, required): I consent to my data being processed for this application
- `cv_file` / `cv_base64` + `cv_filename`: Upload a CV in .pdf or .docx format (<= 10 MB) or provide an `https://` LinkedIn profile URL in `linkedin_url`.

### Alternative: Talent Pool Intake (`join_talent_pool`)

If this mandate is not an exact match, register once in the Talent247 Talent Pool at https://jobs.talent247.co/talent-pool (`https://jobs.talent247.co/talent-pool.md`) via WebMCP tool `join_talent_pool` or `POST https://jobs.talent247.co/api/public/talent-pool`.
