Skip to main content
TALENT247 // SPECIALIST AI EXECUTIVE SEARCH & TECHNICAL RECRUITMENT
ML · Machine Learning

Senior Machine Learning Engineer, Multimodal Evaluations

New York, NY · Full-time · Hybrid

AI Specialism

Machine Learning

Workplace

Hybrid

Location

New York, NY

Compensation

$235k – $320k

annual package

Role Dossier & Specification

REF: SENIOR-MACHINE-LEA

THE CLIENT & MANDATE

Our client is a flagship frontier AI laboratory expanding their East Coast research and engineering hub in New York City. As frontier models unify native speech-to-speech audio, high-resolution vision, video understanding, and autonomous computer control, static text benchmarks fail to capture real-world capability or failure modes—making next-generation multimodal evaluation systems essential to every model training decision.

They have retained Talent247 to hire a Senior Machine Learning Engineer for Multimodal Evaluations in New York, NY. Working directly alongside post-training researchers and product launch leads, the appointed engineer will build interactive, contamination-free evaluation environments measuring real-time voice, visual reasoning, UI navigation, and cross-modal tool use.

KEY RESPONSIBILITIES

  • Architect scalable, low-variance multimodal evaluation pipelines testing native audio prosody/interruption handling, dense chart/diagram spatial reasoning, temporal video grounding, and GUI computer use.
  • Build containerized OS, browser, and code-execution sandboxes where multimodal agents are graded deterministically on end-state environment mutations rather than brittle string matching.
  • Train and calibrate specialized multimodal judge models and rubric verifiers validated rigorously against expert human preference panels.
  • Design automated data-contamination scanners, dynamic benchmark refresh pipelines, and psychometric difficulty-scaling analyses preventing Goodhart overfitting during post-training.
  • Own automated pre-merge and pre-release regression gates running across thousands of checkpoints during active frontier training campaigns.
  • Partner with NYC and SF research teams to turn fine-grained error taxonomies directly into targeted post-training data mixtures.

TECHNICAL ENVIRONMENT & OFFERING

Stack: Python, PyTorch, Rust/TypeScript, containerized VM/browser sandboxes, real-time WebRTC/audio eval harnesses, vLLM/custom inference clusters, Kubernetes. Workplace: Hybrid in New York, NY. Compensation: $235,000 – $320,000 base salary plus top-tier frontier lab equity, comprehensive healthcare, and 401(k).

Core Requirements

ESSENTIAL QUALIFICATIONS

  • 5+ years of machine learning engineering or research engineering experience working on multimodal foundation models, speech/vision systems, or large-scale LLM evaluation infrastructure.
  • Strong hands-on proficiency in PyTorch, distributed batch/streaming inference orchestration, and statistical benchmark design (confidence intervals, power analysis, Elo/Bradley-Terry modeling).
  • Proven ability to architect reliable sandboxed execution harnesses (Docker, Playwright/OS VMs, audio streaming simulators) at high concurrency.
  • Experience diagnosing subtle failure modes across vision encoders, audio tokenizers, OCR/grounding coordinates, or multi-turn agent trajectories.
  • Crisp engineering discipline writing maintainable, reproducible evaluation frameworks trusted for flagship launch decisions.

PREFERRED / BONUS EXPERIENCE

  • Prior authorship of widely cited multimodal, coding, or computer-use benchmarks or evaluation papers at CVPR, NeurIPS, ICML, ICLR, or Interspeech.
  • Experience with full-duplex speech-to-speech evaluation metrics (turn-taking latency, acoustic emotion faithfulness, barge-in robustness).
  • Familiarity with RLVR grader hardening against reward hacking during online RL.

Position Telemetry

Location New York, NY (Hybrid)
Department Machine Learning
Employment type Full-time
Workplace type Hybrid
Salary range
$235k – $320k annual package