Principal ML Engineer - Agent Quality & Evaluation

Atlassian

Sydney, Melbourne

Hybrid

AUD 210,000 - 280,000

Full time

6 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Remote work options

Job summary

Atlassian is seeking a Principal Machine Learning Engineer to set the technical direction for AI agent quality in Studio. You will own the agent quality architecture, define how we measure success, and lead efforts to improve reliability and performance of Rovo Agents used in Jira and Confluence.

You will mentor engineers, shape multi-quarter roadmaps, and collaborate with central AI teams to ensure scalable, production-ready evaluation and monitoring. Remote options available in Australia.

Qualifications

  • 6+ years in ML/AI engineering
  • Proven track record of building and shipping LLM-powered or agentic systems
  • Strong evaluation and experimental skills
  • Strong software and data engineering skills
  • Strong product judgment and decision-making between deterministic, classical ML and LLM-based approaches
  • Experience influencing technical direction across teams and mentoring engineers
  • Regular use of AI tools and ability to translate research into production results
  • Sound judgment on trade-offs across quality, latency, cost, reliability, privacy and access control

Responsibilities

  • Define the agent quality architecture: metrics, sampling, evaluation and roll-up of results
  • Design evaluation systems, benchmarks, datasets and observability for agent behavior
  • Lead reproducible experiments and calibrated evaluators for model, prompt and launch decisions
  • Set up regression gates and pre-release checks for agent, prompt and model changes
  • Lead cross-team technical direction with product and data science
  • Mentor engineers through design reviews and hands-on work
  • Contribute to multi-quarter roadmap and collaborate with central AI teams

Skills

ML/AI engineering
LLMs
NLP
Evaluation & measurement
Software engineering
Data pipelines
Production observability
Mentoring

Education

Masters or PhD in CS/ML

Job description

Atlassian is seeking a Principal Machine Learning Engineer to set the technical direction for AI agent quality in Studio. You will own the agent quality architecture, define how we measure success, and lead efforts to improve reliability and performance of Rovo Agents used in Jira and Confluence.

You will mentor engineers, shape multi-quarter roadmaps, and collaborate with central AI teams to ensure scalable, production-ready evaluation and monitoring. Remote options available in Australia.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior ML Engineer: Agent Quality & Evaluation
Senior ML Engineer: Agent Quality & Evaluation

Atlassian Corp. • Sydney

Hybrid
AUD 150,000 - 190,000
Principal ML Engineer — AI Agent Quality Architect
Principal ML Engineer — AI Agent Quality Architect

Atlassian Corp. • Sydney

Hybrid
AUD 190,000 - 270,000
Principal ML Engineer: Agent Quality & AI Systems
Principal ML Engineer: Agent Quality & AI Systems

Unavailable • Sydney

Hybrid
AUD 180,000 - 260,000
Senior ML Engineer - Agent Quality & Evaluation
Senior ML Engineer - Agent Quality & Evaluation

Unavailable • Sydney

Hybrid
AUD 150,000 - 190,000
Principle Machine Learning Engineer
Principle Machine Learning Engineer

Atlassian • Sydney, Melbourne

Hybrid
AUD 210,000 - 280,000
Remote work options
Principle Machine Learning Engineer
Principle Machine Learning Engineer

Atlassian Corp. • Sydney

Hybrid
AUD 190,000 - 270,000
Principle Machine Learning Engineer
Principle Machine Learning Engineer

Unavailable • Sydney

Hybrid
AUD 180,000 - 260,000
Senior Machine Learning Engineer
Senior Machine Learning Engineer

Atlassian Corp. • Sydney

Hybrid
AUD 150,000 - 190,000
Senior Machine Learning Engineer
Senior Machine Learning Engineer

Unavailable • Sydney

Hybrid
AUD 150,000 - 190,000
Senior ML Platform Engineer: Scalable AI Systems
Senior ML Platform Engineer: Scalable AI Systems

Atlassian • Melbourne

Hybrid
AUD 180,000 - 260,000
Health and wellbeing resources
Paid volunteer days
Top locations