Frontline Language Model Engineer – AI Benchmarking

Artificial Analysis

San Francisco (CA)

On-site

USD 140,000 - 190,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Artificial Analysis in San Francisco is seeking a Forward Deployed Engineer to own the day-to-day operation of our language model benchmarking stack and serve as the technical face of Artificial Analysis to the industry's most important labs.

You'll onboard new models to our evaluation pipeline, run and debug benchmarks, and serve as the primary technical point of contact for AI lab customers: explaining results, fielding methodology questions, and resolving endpoint issues in real time.

Qualifications

  • 3+ years of experience in a client-facing technical role such as solutions engineering, support engineering, or technical consulting.
  • Strong Python proficiency with experience in large codebases you didn’t write.
  • Hands-on experience with AI/ML model APIs (OpenAI, Anthropic, Google, Meta, etc.).
  • Excellent debugging skills; ability to trace issues across APIs, data pipelines, and code.
  • Strong written and verbal English communication for explaining concepts to technical stakeholders.
  • Highly responsive and reliable; you take ownership of customer issues.
  • Comfortable with operational, repeatable work and maintaining reliability.
  • High attention to detail and calm under pressure.

Responsibilities

  • Operate and maintain our Python-based language model benchmarking pipeline end-to-end: onboard new models, configure evaluations, execute benchmark runs, and validate results.
  • Debug issues across the stack from API endpoint timeouts to unexpected benchmark outputs and resolve quickly.
  • Serve as the primary technical contact for AI lab customers: communicate results, explain methodology, and troubleshoot integration issues via Slack and video conferencing.
  • Monitor benchmark runs for anomalies, investigate discrepancies, and ensure accuracy of published results.
  • Maintain documentation of processes, known issues, and model-specific configurations.
  • Collaborate with the engineering team to flag pipeline improvements and contribute to process refinements.
  • Stay current with new model releases, API changes, and developments across the language model ecosystem.

Skills

Python
Client-facing experience
AI/ML model APIs
Debugging
English communication
Ownership
Operational mindset
Attention to detail

Tools

Slack
Video conferencing
APIs

Job description

Artificial Analysis in San Francisco is seeking a Forward Deployed Engineer to own the day-to-day operation of our language model benchmarking stack and serve as the technical face of Artificial Analysis to the industry's most important labs.

You'll onboard new models to our evaluation pipeline, run and debug benchmarks, and serve as the primary technical point of contact for AI lab customers: explaining results, fielding methodology questions, and resolving endpoint issues in real time.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Forward Deployed Engineer, Language Models (Onsite) Equity
Forward Deployed Engineer, Language Models (Onsite) Equity

FDE Times • San Francisco (CA)

On-site
USD 150,000 - 230,000
Equity
Forward Deployed Engineer - Language Models
Forward Deployed Engineer - Language Models

Artificial Analysis • San Francisco (CA)

On-site
USD 140,000 - 190,000
Staff Engineer - Language Model Evaluation
Staff Engineer - Language Model Evaluation

Artificial Analysis • San Francisco (CA)

On-site
USD 180,000 - 240,000
Shape AI direction
World‑expert in AI
Work with leading AI teams
Forward Deployed Engineer - Language Models
Forward Deployed Engineer - Language Models

FDE Times • San Francisco (CA)

On-site
USD 150,000 - 230,000
Equity
Field Engineer, AI Labs — Frontier Benchmarking
Field Engineer, AI Labs — Frontier Benchmarking

Turing • New York (NY), San Francisco (CA), Seattle (WA)

Hybrid
USD 175,000 - 232,000
Equity
Performance incentives
Comprehensive benefits
Applied AI Benchmarking Engineer
Applied AI Benchmarking Engineer

Mercor • San Francisco (CA)

On-site
USD 160,000 - 230,000
Health benefits
Meal stipend
Relocation support
+4
Senior AI/ML Engineer — Benchmarking & Analytics Lead
Senior AI/ML Engineer — Benchmarking & Analytics Lead

FDE Times • San Francisco (CA)

On-site
USD 140,000 - 210,000
Equity
On-site office
Competitive salary
Field Engineer, Frontier AI Labs & Benchmarks
Field Engineer, Frontier AI Labs & Benchmarks

FDE Times • New York (NY)

On-site
USD 175,000 - 232,000
Frontier AI Field Engineer: Benchmarking & Data Delivery
Frontier AI Field Engineer: Benchmarking & Data Delivery

Turing • Oregon

Hybrid
USD 175,000 - 232,000
Member of Technical Staff (Language Model Evaluations)
Member of Technical Staff (Language Model Evaluations)

Artificial Analysis • San Francisco (CA)

On-site
USD 180,000 - 240,000
Shape AI direction
World‑expert in AI
Work with leading AI teams