Remote Language Model Benchmarking Engineer – Client-Facing

Artificial Analysis, Inc.; United States; Hybrid; USD 140,000 - 210,000; Full time

Benefits offered by this job

Equity

Job summary

Artificial Analysis, Inc. is hiring a Forward Deployed Engineer to own the day-to-day operation of our language model benchmarking stack, onboarding new models, running evaluations, and serving as the primary technical face to AI lab customers.

You will explain results, field methodology questions, troubleshoot integration issues, and ensure benchmarks run end-to-end with reliability, while staying current with new model releases and API changes.

Qualifications

• 3+ years in a client-facing technical role (solutions engineering, support engineering, technical consulting, etc.)
• Strong Python proficiency with codebases you didn’t write
• Hands-on experience with AI/ML model APIs (OpenAI, Anthropic, Google, Meta)
• Excellent debugging skills across APIs, data pipelines, and code
• Strong written and verbal English communication

Responsibilities

• Operate and maintain the Python-based language model benchmarking pipeline end-to-end.
• Debug issues across the stack and resolve API timeouts and errors quickly.
• Serve as the primary technical contact for AI lab customers; explain results and troubleshoot integration issues.
• Monitor benchmark runs for anomalies and ensure accuracy of published results.
• Maintain documentation of processes, known issues, and model configurations.
• Collaborate with engineering to flag improvements and refine processes.

Skills

Python proficiency

Client-facing

AI/ML model APIs

Debugging

English communication

Job description

Artificial Analysis, Inc. is hiring a Forward Deployed Engineer to own the day-to-day operation of our language model benchmarking stack, onboarding new models, running evaluations, and serving as the primary technical face to AI lab customers.

You will explain results, field methodology questions, troubleshoot integration issues, and ensure benchmarks run end-to-end with reliability, while staying current with new model releases and API changes.