Senior ML Engineer - Agent Quality & Evaluation

Unavailable

Sydney

Hybride

AUD 150 000 - 190 000

Plein temps

Il y a 12 jours

Obtenez une réponse de cet employeur

Votre CV et votre lettre de motivation, adaptés exactement à ce que l’employeur recherche.

Passez les filtres ATS

Résumé du poste

Atlassian is seeking a Senior Machine Learning Engineer to join the Rovo Agents quality team in Studio, Sydney/Melbourne with remote options. You will design and operate evaluation systems for Rovo agents to measure and improve agent performance across Jira and Confluence integrations.

Responsibilities include building offline/online evaluation suites, calibrating judges, and reducing cost and latency across production LLM systems while collaborating with product, data science and central AI

Qualifications

  • Bachelor's or master's in CS or related field or equivalent experience.
  • 4+ years in ML, including designing, evaluating and operating LL-based or agentic systems.
  • Strong coding skills in Python and/or Kotlin for production code.
  • Experience designing experiments, building evaluation datasets/benchmarks, and analyzing model behavior.
  • Experience building ML systems in production to improve quality, reliability, and cost.
  • Regular use of AI tools with judgment to validate outputs and ownership of results.
  • Strong ownership mindset to lead ambiguous technical work from problem definition to delivery.
  • Clear communication to explain ML trade-offs to engineers, PMs and leaders.

Responsabilités

  • Design, build, ship and operate evaluation and optimization systems for Rovo agents.
  • Build offline evaluation suites and benchmarks reflecting realistic agent tasks.
  • Build online quality metrics on production traffic and evaluate tool usage and agent trajectories.
  • Calibrate judges against human ratings and evolve evals as agents improve.
  • Improve quality while reducing cost and latency across production LLM systems.
  • Own projects from problem definition through deployment and ongoing measurement, contribute to design and code reviews.

Connaissances

Python/Kotlin
Production-grade coding
Experiment design
SQL
Cloud data environments
AI tools literacy
Ownership mindset
Clear communication

Formation

Bachelor's or Master's in CS or related

Outils

SQL
Cloud platforms

Description du poste

Atlassian is seeking a Senior Machine Learning Engineer to join the Rovo Agents quality team in Studio, Sydney/Melbourne with remote options. You will design and operate evaluation systems for Rovo agents to measure and improve agent performance across Jira and Confluence integrations.

Responsibilities include building offline/online evaluation suites, calibrating judges, and reducing cost and latency across production LLM systems while collaborating with product, data science and central AI

Obtenez votre examen gratuit et confidentiel de votre CV.

ou faites glisser et déposez votre fichier ici.

Similar jobs

Postes similaires à comparer

Senior ML Engineer: Agent Quality & Evaluation
Senior ML Engineer: Agent Quality & Evaluation

Atlassian Corp. • Sydney

Hybride
AUD 150 000 - 190 000
Principal ML Engineer - Agent Quality & Evaluation
Principal ML Engineer - Agent Quality & Evaluation

Atlassian • Sydney, Melbourne

Hybride
AUD 210 000 - 280 000
Remote work options
Senior Machine Learning Engineer
Senior Machine Learning Engineer

Atlassian Corp. • Sydney

Hybride
AUD 150 000 - 190 000
Senior Machine Learning Engineer
Senior Machine Learning Engineer

Unavailable • Sydney

Hybride
AUD 150 000 - 190 000
Principal ML Engineer: Agent Quality & AI Systems
Principal ML Engineer: Agent Quality & AI Systems

Unavailable • Sydney

Hybride
AUD 180 000 - 260 000
Principal ML Engineer — AI Agent Quality Architect
Principal ML Engineer — AI Agent Quality Architect

Atlassian Corp. • Sydney

Hybride
AUD 190 000 - 270 000
Principle Machine Learning Engineer
Principle Machine Learning Engineer

Atlassian • Sydney, Melbourne

Hybride
AUD 210 000 - 280 000
Remote work options
Principle Machine Learning Engineer
Principle Machine Learning Engineer

Atlassian Corp. • Sydney

Hybride
AUD 190 000 - 270 000
Principle Machine Learning Engineer
Principle Machine Learning Engineer

Unavailable • Sydney

Hybride
AUD 180 000 - 260 000
Senior ML Platform Engineer: Scalable AI Systems
Senior ML Platform Engineer: Scalable AI Systems

Atlassian • Melbourne

Hybride
AUD 180 000 - 260 000
Health and wellbeing resources
Paid volunteer days
Lieux principaux