Skip to main content
TALENT247 // SPECIALIST AI EXECUTIVE SEARCH & TECHNICAL RECRUITMENT
ML · Machine Learning

Machine Learning Engineer, Frontier Safety & Red-Team Evals

San Francisco, CA · Full-time · Hybrid

AI Specialism

Machine Learning

Workplace

Hybrid

Location

San Francisco, CA

Compensation

$190k – $270k

annual package

Role Dossier & Specification

REF: MACHINE-LEARNING-E

THE CLIENT & MANDATE

Our client is a premier San Francisco AI data, post-training, and evaluation leader operating independent frontier leaderboards and pre-deployment safety testing partnerships with leading commercial labs and national AI safety institutes. Their Frontier Safety & Evaluations group designs ungameable private benchmarks and automated red-teaming systems that measure catastrophic dual-use risks, autonomous cyber capabilities, and deceptive agent behaviors before public model release.

They have partnered with Talent247 to recruit a Machine Learning Engineer for Frontier Safety & Red-Team Evals in San Francisco. Combining psychometric rigor, adversarial ML, and sandboxed agent harnesses, the successful candidate will build the gold-standard safety measurement stack relied upon across the global AI ecosystem.

KEY RESPONSIBILITIES

  • Architect private, contamination-resistant multimodal and agentic safety benchmarks evaluating cyber-offense uplift, biosecurity/chemical dual-use risk, persuasion, autonomous replication, and scheming.
  • Build multi-turn automated red-teaming swarms that adaptively discover multi-step jailbreaks, tool-use privilege escalations, and indirect prompt injections across frontier APIs and open-weight checkpoints.
  • Apply Item Response Theory (IRT), Bradley-Terry-Luce ranking models, and bootstrap confidence estimation to turn noisy human and automated red-team trials into statistically calibrated risk scores.
  • Engineer air-gapped, network-jailed Capture-the-Flag (CTF) and vulnerable-application sandboxes to safely measure autonomous vulnerability discovery and exploit generation.
  • Train specialized safety judge models and rubric verifiers calibrated against PhD domain-expert human adjudicators.
  • Co-author influential public technical reports and private pre-launch safety audits delivered to frontier model developers and regulatory partners.

TECHNICAL ENVIRONMENT & OFFERING

Stack: Python, PyTorch, Inspect-style agent eval frameworks, Docker/microVM sandboxes, vLLM, statistical modeling (PyMC/Stan/SciPy), Kubernetes. Workplace: Hybrid in San Francisco, CA. Compensation: $190,000 – $270,000 base salary plus equity participation and comprehensive US benefits.

Core Requirements

ESSENTIAL QUALIFICATIONS

  • 3+ years of machine learning engineering, AI safety research engineering, or adversarial security ML experience.
  • Proven track record designing LLM/agent evaluation harnesses, adversarial red-teaming pipelines, or fine-tuning classifier/judge models in PyTorch.
  • Solid statistical grounding in benchmark design, inter-annotator agreement, variance reduction, contamination auditing, and ranking calibration.
  • Strong Python software engineering skills and familiarity with containerized sandbox isolation for untrusted model-generated code execution.
  • High integrity and security mindset suitable for handling sensitive pre-release frontier model weights and dual-use evaluation datasets.

PREFERRED / BONUS EXPERIENCE

  • Background in offensive cybersecurity (CTF competitions, vulnerability research), computational biology/chemistry, or AI alignment evaluations.
  • Publications or open-source contributions in AI safety benchmarks, jailbreak robustness, or evaluation methodology.
  • Experience evaluating multimodal (voice/vision) or long-horizon computer-use agents.

Position Telemetry

Location San Francisco, CA (Hybrid)
Department Machine Learning
Employment type Full-time
Workplace type Hybrid
Salary range
$190k – $270k annual package