Principal AI Agent Quality Architect

Atlassian

United States

Remote

USD 210,000 - 260,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Atlassian is seeking a Principal Machine Learning Engineer to set the technical direction for AI agent quality in Studio. This is a hands-on leadership role focused on defining metrics, sampling, and evaluation for agents built on our platform.

You will lead architecture, experiments, and governance across a multi-team effort with remote options. You’ll mentor engineers, design benchmarks, and drive measurable improvements in agent quality across production deployments, including LLM-powered

Qualifications

  • 6+ years in ML/AI engineering with deep expertise in LLMs, NLP, information retrieval or recommendation.
  • Proven track record shipping LLM-powered or agentic systems used by real customers.
  • Strong evaluation and experimental skills to design benchmarks and assess quality.
  • Strong software and data engineering skills with data pipelines, model serving and observability.
  • Strong product judgment: define measurable outcomes and choose ML approaches.
  • Experience influencing technical direction across teams and mentoring engineers.
  • Regular use of AI tools and translating research into production results.
  • Sound judgment on quality, latency, cost, reliability, privacy and access control.

Responsibilities

  • Own the agent quality architecture.
  • Define what we measure, how production traffic is sampled and judged.
  • Design for cost and accuracy.
  • Decide which quality questions need deterministic checks, heuristics, or model-based judgment.
  • Build the evaluation system around those decisions.
  • Architect evaluation and experimentation.
  • Define benchmarks, datasets, evaluation environments and observability.
  • Guide teams to build reproducible experiments and calibrated evaluators.
  • Make quality part of how we ship.
  • Set up regression gates and pre-release checks for changes.
  • Lead hardest quality problems hands-on (agent selection, tool-use reliability).
  • Lead technical direction across teams and multi-quarter roadmaps.
  • Grow ML craft; mentor engineers in a team of software engineers.

Skills

LLMs
NLP
Information Retrieval
Recommendation Systems
Production Observability
Data Pipelines
Model Serving
Mentoring
Leadership

Education

Master's or PhD in CS/ML

Job description

Atlassian is seeking a Principal Machine Learning Engineer to set the technical direction for AI agent quality in Studio. This is a hands-on leadership role focused on defining metrics, sampling, and evaluation for agents built on our platform.

You will lead architecture, experiments, and governance across a multi-team effort with remote options. You’ll mentor engineers, design benchmarks, and drive measurable improvements in agent quality across production deployments, including LLM-powered

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Principle Machine Learning Engineer
Principle Machine Learning Engineer

Atlassian • United States

Remote
USD 210,000 - 260,000
Senior Principal AI Platform Engineer
Senior Principal AI Platform Engineer

Atlassian Corp. • United States

On-site
USD 220,000 - 287,000
Benefits package
Principal ML Systems Engineer - Scalable AI & Evaluation
Principal ML Systems Engineer - Scalable AI & Evaluation

Atlassian Corp. • San Francisco (CA)

Remote
USD 237,000 - 309,000
Health and wellbeing resources
Volunteer days
Equity and benefits
Senior Machine Learning Architect - AI Systems & Strategy
Senior Machine Learning Architect - AI Systems & Strategy

Atlassian • San Francisco (CA)

Remote
USD 237,000 - 309,000
Health benefits
Paid volunteer days
Flexible work arrangements
Lead AI Factory Engineer - Architect & Verifier of Quality Code
Lead AI Factory Engineer - Architect & Verifier of Quality Code

EPAM Systems • United States

Remote
USD 140,000 - 240,000
ML Engineer, AI Assistant Quality — Production Systems
ML Engineer, AI Assistant Quality — Production Systems

FDE Times • San Francisco (CA)

Hybrid
USD 180,000 - 205,000
Home office stipend
Education stipend
Wellness stipend
+1
AI Factory Architect & Code Quality Lead
AI Factory Architect & Code Quality Lead

EPAM Systems • United States

Remote
USD 180,000 - 250,000
Principal AI Systems Architect – Agent Orchestration
Principal AI Systems Architect – Agent Orchestration

Momentum • Massachusetts

On-site
USD 180,000 - 270,000
Director of AI Quality & Compliance
Director of AI Quality & Compliance

LILT AI • City of Syracuse (NY)

Hybrid
USD 180,000 - 250,000
Principal AI Engineer: LLMs & Agentic Production
Principal AI Engineer: LLMs & Agentic Production

FactSet • New York (NY)

On-site
USD 180,000 - 235,000
Health insurance
Life insurance
Disability insurance
+5