Senior ML Engineer: Agent Quality & Evaluation

Atlassian Corp.; Sydney; Hybrid; AUD 150,000 - 190,000; Full time

Job summary

Atlassian seeks a Senior Machine Learning Engineer to join the Rovo Agents quality team in Studio. You will design, build and operate evaluation systems to measure and improve AI agents across Jira, Confluence and automation scenarios.

You will work with a team in Sydney and Melbourne, with remote options, and collaborate with product, data science and central AI teams to advance production ML for real-world agentic systems.

Qualifications

• Degree in CS or related field or equivalent experience.
• 4+ years in ML industry, especially with LLM-based or agentic systems.
• Hands-on coding in Python/Kotlin for production-grade systems.
• Experience designing experiments, datasets and benchmarks for ML.
• Familiarity with AI tooling and evaluating model behavior.
• Ability to own and drive technical work end-to-end.
• Strong communication to non-engineers.

Responsibilities

• Design, build, ship and operate evaluation and optimization systems for Rovo agents.
• Build offline evaluation suites and benchmarks reflecting realistic agent tasks.
• Build online quality metrics on production traffic and analyze tool errors and sessions.
• Calibrate judges with human ratings and independent signals to avoid saturation.
• Improve agent quality across prompts, tool use, retrieval and routing.
• Own projects from problem definition through deployment and ongoing measurement.

Skills

Python

Kotlin

Experiment design

Evaluation & benchmarking

Tooling & production code

AI tool usage

Ownership mindset

Clear communication

Education

Bachelor's or Master's in CS or related field

Tools

SQL

Distributed data processing

Cloud data environments

Job description

Atlassian seeks a Senior Machine Learning Engineer to join the Rovo Agents quality team in Studio. You will design, build and operate evaluation systems to measure and improve AI agents across Jira, Confluence and automation scenarios.

You will work with a team in Sydney and Melbourne, with remote options, and collaborate with product, data science and central AI teams to advance production ML for real-world agentic systems.