Language Model Benchmarking Engineer – On‑Site, Frontline

Artificial Analysis Inc; City of Melbourne; On-site; AUD 140,000 - 210,000; Full time

Benefits offered by this job

Equity

Job summary

Artificial Analysis is seeking a Forward Deployed Engineer to own the day-to-day operation of our language model benchmarking stack in Melbourne (ONSITE 3 days/week) and SF (ONSITE 5 days/week). You will onboard models, run benchmarks, and serve as the primary technical contact for AI labs and enterprise customers, explaining results, resolving endpoint issues, and ensuring reliability.

You will collaborate with engineering to improve pipelines, stay current with API changes, and maintain clear

Qualifications

• 3+ years in a client-facing technical role in a fast-moving tech company.
• Strong Python proficiency and ability to navigate large codebases.
• Hands-on experience with AI/ML model APIs (OpenAI, Anthropic, Google, Meta, etc.).
• Excellent debugging skills to trace issues across APIs, data pipelines and code.
• Strong written and spoken English for communicating with technical stakeholders.
• Ownership mindset and reliability; able to own issues end-to-end.
• Comfort with repeatable, operations-style work; detail-oriented and calm under pressure.

Responsibilities

• Onboard new models, configure evaluations, run benchmarks, and validate results.
• Debug issues across API endpoints, data pipelines, and benchmark outputs.
• Be the primary technical contact for AI lab customers; explain results and resolve issues.
• Monitor runs for anomalies and ensure accuracy of published results.
• Maintain documentation and model-specific configurations; collaborate on pipeline improvements.
• Stay current with new model releases and API changes.

Skills

Python

AI APIs

Debugging

Client-facing

English communication

Tools

OpenAI API

Anthropic API

Google AI API

Job description

Artificial Analysis is seeking a Forward Deployed Engineer to own the day-to-day operation of our language model benchmarking stack in Melbourne (ONSITE 3 days/week) and SF (ONSITE 5 days/week). You will onboard models, run benchmarks, and serve as the primary technical contact for AI labs and enterprise customers, explaining results, resolving endpoint issues, and ensuring reliability.

You will collaborate with engineering to improve pipelines, stay current with API changes, and maintain clear