Remote Principal ML Engineer: Agent Quality Architect

Atlassian; Sydney; Hybrid; AUD 180,000 - 230,000; Full time

Job summary

Atlassian is seeking a Principal Machine Learning Engineer to set the technical direction for AI agent quality in Studio. This hands-on leadership role will define how we measure agent quality, sampling production traffic, and drive improvements for agents built on our platform.

Based in Sydney/Melbourne with remote options, you will own the agent quality architecture, design evaluation systems, and lead multi-team efforts to ship higher quality agents that perform reliably across Jira and

Qualifications

• 6+ years in ML/AI engineering with deep expertise in LLMs, NLP, information retrieval or recommendation.
• Proven track record shipping LLM-powered or agentic systems used by real customers.
• Strong evaluation and experimental skills for benchmarks and offline-to-product mapping.
• Strong software and data engineering skills with production observability.
• Strong product judgment to define measurable outcomes and select approaches.
• Experience influencing technical direction across teams and mentoring senior engineers.
• Regular use of AI tools and turning research into production results.
• Sound judgment on quality, latency, cost, reliability, privacy and access control.

Responsibilities

• Own the agent quality architecture and define measurement and reporting.
• Design evaluation systems balancing determinism, heuristics and model-based judgment.
• Architect evaluation and experimentation with benchmarks and observability.
• Implement regression gates and pre-release checks for agent and model changes.
• Lead hardest quality problems hands-on including tool-use and routing.
• Set multi-quarter roadmaps with product and data science teams.
• Grow ML craft by mentoring engineers through reviews and hands-on work.

Skills

ML/AI engineering

LLMs / NLP

Evaluation & experimentation

Software & data engineering

Product judgment

Technical leadership

AI tooling

Trade-off judgment

Job description

Atlassian is seeking a Principal Machine Learning Engineer to set the technical direction for AI agent quality in Studio. This hands-on leadership role will define how we measure agent quality, sampling production traffic, and drive improvements for agents built on our platform.

Based in Sydney/Melbourne with remote options, you will own the agent quality architecture, design evaluation systems, and lead multi-team efforts to ship higher quality agents that perform reliably across Jira and