Principal ML Engineer: Agent Quality & AI Systems

Unavailable

Sydney

Hybrid

AUD 180,000 - 260,000

Full time

12 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Atlassian is seeking a Principal Machine Learning Engineer to set the technical direction for AI agent quality in Studio. This hands-on leadership role defines how we measure agent quality and leads efforts to improve it, across production systems.

The team operates in Sydney and Melbourne with remote options, collaborating with product, data science and central AI teams to advance applied ML on real agentic systems in production.

Qualifications

  • 6+ years in ML/AI engineering with deep expertise in LLMs, NLP, information retrieval or recommendation.
  • Proven track record of building and shipping LLM-powered or agentic systems used by real customers.
  • Strong evaluation and experimental skills with benchmarks and leakage detection.
  • Strong software and data engineering skills with data pipelines and production observability.
  • Strong product judgment to choose between deterministic, classical ML and LLM-based approaches.
  • History of influencing technical direction across multiple teams and mentoring engineers.
  • Regular use of AI tools and translating research into production results.
  • Sound judgment on trade-offs between quality, latency, cost, reliability, privacy.

Responsibilities

  • Own the agent quality architecture: define measurement, sampling, and roll-up of results.
  • Design for cost and accuracy: determine deterministic checks vs. heuristics vs. model-based judgments.
  • Architect evaluation and experimentation: benchmarks, datasets, evaluation environments, observability.
  • Make quality part of shipping: regression gates, pre-release checks, drift detection.
  • Lead the hardest quality problems hands-on (e.g., agent routing, tool-use reliability).
  • Set multi-quarter roadmap with product and data science, align with central AI teams.
  • Grow ML craft: mentor engineers through design reviews and hands-on work.

Skills

ML/AI engineering
LLMs/NLP
Evaluation & experimentation
Software & data engineering
Product judgment
Cross-team influence
AI tooling proficiency
Production observability

Education

Master’s or PhD in Computer Science / ML or related field

Job description

Atlassian is seeking a Principal Machine Learning Engineer to set the technical direction for AI agent quality in Studio. This hands-on leadership role defines how we measure agent quality and leads efforts to improve it, across production systems.

The team operates in Sydney and Melbourne with remote options, collaborating with product, data science and central AI teams to advance applied ML on real agentic systems in production.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Principal ML Engineer — AI Agent Quality Architect
Principal ML Engineer — AI Agent Quality Architect

Atlassian Corp. • Sydney

Hybrid
AUD 190,000 - 270,000
Principal ML Engineer - Agent Quality & Evaluation
Principal ML Engineer - Agent Quality & Evaluation

Atlassian • Sydney, Melbourne

Hybrid
AUD 210,000 - 280,000
Remote work options
Senior ML Engineer: Agent Quality & Evaluation
Senior ML Engineer: Agent Quality & Evaluation

Atlassian Corp. • Sydney

Hybrid
AUD 150,000 - 190,000
Principle Machine Learning Engineer
Principle Machine Learning Engineer

Atlassian Corp. • Sydney

Hybrid
AUD 190,000 - 270,000
Senior ML Engineer - Agent Quality & Evaluation
Senior ML Engineer - Agent Quality & Evaluation

Unavailable • Sydney

Hybrid
AUD 150,000 - 190,000
Principle Machine Learning Engineer
Principle Machine Learning Engineer

Unavailable • Sydney

Hybrid
AUD 180,000 - 260,000
Principle Machine Learning Engineer
Principle Machine Learning Engineer

Atlassian • Sydney, Melbourne

Hybrid
AUD 210,000 - 280,000
Remote work options
Principal Ml Engineer – Ai Infrastructure & Systems
Principal Ml Engineer – Ai Infrastructure & Systems

Atlassian • Sydney

Remote
AUD 200,000 - 260,000
Senior Machine Learning Engineer
Senior Machine Learning Engineer

Atlassian Corp. • Sydney

Hybrid
AUD 150,000 - 190,000
Senior Machine Learning Engineer
Senior Machine Learning Engineer

Unavailable • Sydney

Hybrid
AUD 150,000 - 190,000
Top locations