Principal ML Engineer — AI Agent Quality Architect

Atlassian Corp.; Sydney; Hybrid; AUD 190,000 - 270,000; Full time

Job summary

Atlassian Corp. is seeking a Principal Machine Learning Engineer to set the technical direction for AI agent quality in Studio.

This hands-on leadership role will define how we measure agent quality and lead efforts to improve performance in production. The role requires shaping evaluation frameworks, balancing cost and accuracy, and guiding multi-team efforts across Sydney and Melbourne with remote options.

Qualifications

• 6+ years in ML/AI engineering with deep expertise in LLMs, NLP, information retrieval or recommendation.
• Proven track record shipping LLM-powered or agentic systems used by real customers, and improving them over time through measurement.
• Strong evaluation and experimental skills; design useful benchmarks, assess label and judge quality, and link offline results to product impact.
• Strong software and data engineering skills; validate designs in code and work with data pipelines, model serving and production observability.
• Strong product judgment: define measurable outcomes and choose between deterministic, classical ML and LLM-based approaches.
• A track record of influencing technical direction across teams and mentoring senior engineers.
• Regular use of AI tools in your own work, turning new research into production results.
• Sound judgment about trade-offs across quality, latency, cost, reliability, privacy and access control.

Responsibilities

• Own the agent quality architecture; define what to measure and how results roll up.
• Design for cost and accuracy; decide which checks are deterministic or heuristic.
• Architect evaluation and experimentation; define benchmarks and observability for agent behavior.
• Make quality part of shipping; set up gates and drift detection for production traffic.
• Lead the hardest quality problems hands-on, including tool-use reliability and agent routing.
• Lead technical direction across teams; set multi-quarter roadmaps with product and data science.
• Grow ML craft; mentor engineers through design reviews and hands-on work.

Skills

LLMs

NLP

Information retrieval

Recommendation

Evaluation

Benchmarks

Data pipelines

Model serving

Observability

Mentoring

AI tooling

Research-to-production

Education

Master's or PhD in CS/ML or related field

Job description

Atlassian Corp. is seeking a Principal Machine Learning Engineer to set the technical direction for AI agent quality in Studio.

This hands-on leadership role will define how we measure agent quality and lead efforts to improve performance in production. The role requires shaping evaluation frameworks, balancing cost and accuracy, and guiding multi-team efforts across Sydney and Melbourne with remote options.