Skip to main content
TALENT247 // SPECIALIST AI EXECUTIVE SEARCH & TECHNICAL RECRUITMENT
ML · Machine Learning

Staff Machine Learning Engineer, Agent Evaluation & Judges

San Francisco, CA · Full-time · Hybrid

AI Specialism

Machine Learning

Workplace

Hybrid

Location

San Francisco, CA

Compensation

$235k – $330k

annual package

Role Dossier & Specification

REF: STAFF-MACHINE-LEAR

THE CLIENT & MANDATE

Our client is a globally scaled data and artificial intelligence platform leader headquartered in San Francisco, empowering thousands of enterprises to build, evaluate, govern, and serve custom agentic AI systems over their proprietary data estates. Their AI Research & Agent Evaluation organization builds the automated judge models, synthetic benchmark generators, and reinforcement-learning feedback loops that allow enterprise engineering teams to hill-climb compound agent quality systematically without manual labeling bottlenecks.

They have retained Talent247 to appoint a Staff Machine Learning Engineer for Agent Evaluation & Judges in San Francisco, CA. Combining post-training research depth with large-scale product impact, the successful candidate will architect custom generative judge models, automatic prompt/weight optimizers, and trajectory-diagnosis algorithms deployed across tens of thousands of enterprise AI pipelines.

KEY RESPONSIBILITIES

  • Train and post-train specialized, bias-mitigated generative LLM-as-a-judge and process-reward models that evaluate multi-turn tool-calling trajectories, retrieval groundedness, SQL/code correctness, and domain policy compliance.
  • Eliminate classic judge failure modes—position bias, verbosity bias, self-enhancement bias, and rubric drift—through principled preference alignment, chain-of-critique post-training, and small-sample human calibration adapts.
  • Pioneer automated synthetic test-set synthesis and adversarial edge-case mining pipelines generating realistic multi-hop evaluation benchmarks directly from enterprise document/table schemas and production trace logs.
  • Co-design closed-loop agent optimization algorithms (automated prompt/rubric evolution, teacher-student distillation, and online RLAIF/GRPO) that turn evaluation failures directly into measurable model upgrades.
  • Partner with core platform engineering teams to ship sub-100ms streaming guardrail judges and asynchronous batch trace evaluators inside unified developer SDKs and observability UIs.
  • Publish influential research and open benchmarks advancing reliable evaluation of enterprise compound AI systems.

TECHNICAL ENVIRONMENT & OFFERING

Stack: Python, PyTorch, FSDP/Megatron, Verl/TRL, vLLM/SGLang, Ray, Spark, OpenTelemetry agent tracing, AWS/GCP/Azure GPU clusters. Workplace: Hybrid in San Francisco, CA. Compensation: $235,000 – $330,000 base salary plus substantial equity grants, annual refreshers/bonus, and comprehensive US benefits.

Core Requirements

ESSENTIAL QUALIFICATIONS

  • 7+ years of machine learning engineering or applied AI research experience, including Staff-level technical leadership in LLM post-training, evaluation systems, reward modeling, or compound AI optimization.
  • Deep hands-on track record training custom language models (SFT, DPO, RLHF/RLAIF, GRPO) and building statistically rigorous LLM evaluation frameworks in PyTorch.
  • Strong command of psychometric/statistical calibration, inter-rater reliability, active learning from sparse human feedback, and causal error attribution in multi-step agent graphs.
  • Proven ability to translate frontier ML research into widely adopted developer frameworks or enterprise cloud platform products.
  • Track record mentoring senior ML engineers and setting multi-quarter technical strategy across research and product engineering pods.

PREFERRED / BONUS EXPERIENCE

  • Publications at NeurIPS, ICML, ICLR, ACL, or EMNLP on LLM judges, reward modeling, automated prompt/agent optimization (DSPy/TextGrad-class paradigms), or synthetic data generation.
  • Experience optimizing compact 3B–14B judge/verifier models for high-throughput, low-cost production trace scoring.
  • Core contributions to major open-source AI evaluation, tracing, or agent lifecycle frameworks.

Position Telemetry

Location San Francisco, CA (Hybrid)
Department Machine Learning
Employment type Full-time
Workplace type Hybrid
Salary range
$235k – $330k annual package