Senior AI Researcher: Agent Evaluation & Performance

Cobalt; New York (NY); On-site; USD 160,000 - 230,000; Full time

Job summary

Cobalt is seeking an experienced Senior AI/ML researcher to lead evaluation and benchmarking of its agent ecosystem, including code Q&A and MCP tooling. You will design eval harnesses, ground truth from production data, and measure recall, precision, and efficiency across models and prompts.

The role requires 7+ years in ML/NLP research, deep knowledge of agent architectures, and production-grade Python tooling.

Qualifications

• 7+ years in ML/NLP research or applied AI with evaluating LLMs or agents.
• Experience designing benchmarks: task sampling, contamination control, scoring methods, and power analysis.
• Deep knowledge of agent architectures: tool use, retrieval, multi-step planning, MCP or similar protocols.
• Expert Python, and production-quality eval infrastructure you have shipped.
• Experience leading research projects end to end, from design to publication or product.

Responsibilities

• Own the evaluation function for all Cobalt agents: code Q&A, change-impact analysis, incident response, and MCP tool use by coding agents.
• Design and build eval harnesses and benchmarks, with ground truth taken from merged PRs, production traces, and code.
• Define and track agent performance metrics: recall and precision, cost per correct answer, tool-call efficiency, latency, and failure modes.
• Run controlled comparisons across models, prompts, retrieval strategies, and baselines, with statistical rigor.
• Diagnose agent failures in retrieval, reasoning, and tool design, and drive the fixes with engineering.
• Build regression suites in CI that score every model, prompt, and connector change before release.
• Lead external research, including ownership of research papers.
• Set the research agenda and mentor future hires.

Skills

7+ years in ML/NLP research or applied

Benchmarks design

Agent architectures knowledge

Expert Python

Experience leading research projects

Tools

Python

CI/CD pipelines

Experiment tracking

Job description

Cobalt is seeking an experienced Senior AI/ML researcher to lead evaluation and benchmarking of its agent ecosystem, including code Q&A and MCP tooling. You will design eval harnesses, ground truth from production data, and measure recall, precision, and efficiency across models and prompts.

The role requires 7+ years in ML/NLP research, deep knowledge of agent architectures, and production-grade Python tooling.