Senior AI Quality Developer, Personalization

Autodesk, Inc

Canada

Remote

CAD 84,000 - 123,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Autodesk, Inc. is seeking a Senior AI Quality Developer in Canada to design and implement quality engineering for AI-powered personalization and RAG-driven experiences.

You will build scalable test frameworks and evaluation pipelines, collaborating with applied scientists, developers, and product teams to ensure reliable, trustworthy AI software across Autodesk Assistant and related products. You will work on nondeterministic AI evaluation methods, ensure robust test coverage, and contribute to

Qualifications

  • BS or MS in Computer Science, Software Engineering, or a related field, or equivalent practical experience.
  • 6+ years of professional software quality engineering, test automation, or related disciplines.
  • Strong hands-on programming with TypeScript and Python, including scalable automated testing frameworks.

Responsibilities

  • Design quality engineering strategies for AI-powered personalization and RAG-based systems.
  • Build and maintain automated testing frameworks for web apps and cloud services.
  • Develop golden datasets, benchmarks, and evaluation pipelines for nondeterministic AI behavior.
  • Evaluate RAG and personalization systems for retrieval quality and relevance.
  • Test agentic workflows including tool selection, memory, and error handling.
  • Design automated checks for modular, distributed systems and API contracts.
  • Integrate tests and AI evaluations into CI/CD pipelines and release gates.
  • Collaborate with scientists and developers to identify AI failure modes and improve reliability.

Skills

TypeScript
Python
Playwright
Puppeteer
Pytest
AI testing
LLMs
Cloud
CI/CD
Observability

Education

BS or MS in CS or related field, or equivalent practical experience

Tools

Playwright
Puppeteer
Pytest
AWS

Job description

Position Overview

At Autodesk, we're exploring new ways to make design and engineering software smarter, more intuitive, and more personal. AI is helping us rethink how millions of people interact with products like AutoCAD, Revit, Autodesk Construction Cloud, and Forma. Our Personalized Experiences Team works across Autodesk to bring AI, machine learning, and personalization into our products. We collaborate with product teams, experiment with new ideas, and build capabilities that benefit customers across the Autodesk ecosystem. We're looking for a Senior AI Quality Developer to help make those experiences reliable and trustworthy. In this role, you'll work on Autodesk Assistant, helping ensure we deliver high-quality, AI-powered experiences to our users. AI opens up exciting possibilities but also brings new challenges in how we test and evaluate software. You'll write code, build automation, solve interesting problems, and explore better ways to approach quality alongside software developers, applied scientists, and product teams. More info about Autodesk Assistant: https://www.autodesk.com/solutions/autodesk-ai/autodesk-assistant

Responsibilities
  • Design and implement quality engineering strategies for AI-powered personalization, recommendation systems, LLM applications, retrieval-augmented generation (RAG), and agentic systems
  • Build and maintain scalable automated functional, integration, API, end-to-end, and performance testing frameworks using TypeScript, Python, Playwright, Puppeteer, Pytest, or equivalent tools across web applications and cloud services
  • Develop golden datasets, benchmark suites, regression tests, and automated evaluation pipelines for nondeterministic AI behavior, including rubric-based, statistical, and LLM-as-a-judge evaluation methods
  • Evaluate RAG and personalization systems for retrieval quality, groundedness, relevance, ranking accuracy, consistency, and response quality across different user contexts
  • Test agentic workflows, including tool selection and execution, multistep orchestration, memory, error handling, authorization boundaries, and end-to-end task completion
  • Design automated quality checks for modular and distributed systems, including API contracts, integration boundaries, component compatibility, architectural regressions, and service reliability
  • Integrate automated tests and AI evaluations into CI/CD pipelines, establishing quality gates and release readiness criteria
  • Collaborate with applied scientists and developers to identify AI failure modes, evaluate security and privacy risks, and improve reliability, performance, observability, and production quality metrics
  • Champion quality engineering best practices, leverage AI-assisted development tools responsibly, and mentor developers across cross-functional teams
Minimum Qualifications
  • BS or MS in Computer Science, Software Engineering, or a related field, or equivalent practical experience 6 or more years of professional experience in software engineering, quality engineering, test automation, or related disciplines
  • Strong hands-on programming experience with TypeScript and Python, including designing and maintaining scalable automated testing frameworks
  • Experience with browser automation using Playwright, Puppeteer, or equivalent tools, and Python-based testing using Pytest or equivalent frameworks, covering functional, API, integration, and end-to-end testing
  • Practical experience testing or evaluating AI/ML-powered applications, including LLM-based systems, and developing automated evaluations for correctness, relevance, consistency, and reliability
  • Understanding of LLMs, embeddings, RAG, and AI agent architectures, including common failure modes, nondeterministic behavior, evaluation datasets, and regression testing approaches
  • Experience testing complex cloud services, distributed systems, or modular applications, including API contracts, integration boundaries, debugging, and system reliability
  • Experience with CI/CD pipelines, cloud environments (AWS preferred), performance testing, and production observability using logs, metrics, and traces
  • Understanding of data validation, security, privacy, and Responsible AI considerations when testing AI-powered systems
  • Ability to work independently, communicate technical findings clearly, and collaborate effectively with applied scientists, developers, and product partners
Preferred Qualifications
  • Experience with AI evaluation tools such as DeepEval, Ragas, LangSmith, promptfoo, Braintrust, or similar frameworks, including benchmark design, LLM-as-a-judge, and statistical or human evaluation methods
  • Experience testing agentic applications built using LangGraph, LangChain, or comparable orchestration frameworks, including adversarial testing and AI safety evaluation
  • Experience evaluating RAG systems using retrieval metrics, alongside groundedness, faithfulness, and answer quality
  • Experience with recommendation systems, ranking models, personalization, or search relevance, including offline and online experimentation and A/B testing
  • Familiarity with MLOps, model monitoring, dataset versioning, data validation pipelines, and production AI evaluation workflows
  • Experience evaluating AI inference performance, scalability, latency, throughput, and resource utilization in distributed or event-driven systems
  • Experience testing extensible or plugin-based applications, including API compatibility, SDKs, extension mechanisms, dependency validation, and consumer-driven contract testing
  • Experience using AI coding agents to develop reliable test automation, mentoring developers, or leading cross-team quality initiatives
Learn More / Plus d'information
About Autodesk / À propos d’Autodesk

Welcome to Autodesk! Amazing things are created every day with our software – from the greenest buildings and cleanest cars to the smartest factories and biggest hit movies. We help innovators turn their ideas into reality, transforming not only how things are made, but what can be made. We take great pride in our culture here at Autodesk – it’s at the core of everything we do. Our culture guides the way we work and treat each other, informs how we connect with customers and partners, and defines how we show up in the world. When you’re an Autodesker, you can do meaningful work that helps build a better world designed and made for all. Ready to shape the world and your future? Join us! Bienvenue chez Autodesk ! Nos logiciels créent chaque jour des choses extraordinaires : des bâtiments les plus écologiques aux voitures les plus propres, en passant par les usines les plus intelligentes et les films à succès. Nous aidons les innovateurs à concrétiser leurs idées, transformant non seulement la façon dont les choses sont fabriquées, mais aussi ce qui peut l’être. Nous sommes très fiers de notre culture chez Autodesk ; elle est au cœur de tout ce que nous faisons. Notre culture guide notre façon de travailler et de nous comporter les uns envers les autres, influence nos interactions avec nos clients et nos partenaires, et définit notre image au monde. En tant qu’Autodeskien, vous pouvez accomplir un travail significatif qui contribue à bâtir un monde meilleur, conçu et réalisé pour tous. Prêt à façonner le monde et votre avenir ? Rejoignez-nous !

Salary transparency / Transparence Salariale

Salary is one part of Autodesk’s competitive compensation package. For Canada based roles, we expect a starting base salary between $84,000 and $123,200. Offers are based on the candidate’s experience and geographic location, and may exceed this range. In addition to base salaries, our compensation package may include annual cash bonuses, commissions for sales roles, stock grants, and a comprehensive benefits package. Le salaire est l'un des éléments du programme de rémunération concurrentiel d'Autodesk. Pour les postes basés au Canada, nous offrons un salaire de base entre $84,000 et $123,200. Le salaire est déterminé selon l'expérience professionnelle et l'emplacement du candidat(e), et peut dépasser cette fourchette. En plus du salaire de base, notre programme de rémunération peut inclure une prime annuelle, des commissions pour les postes de ventes, des attributions d'actions et un ensemble complet d'avantages sociaux.

Belonging

We take pride in cultivating a culture of belonging where everyone can thrive. Learn more here: https://www.autodesk.com/company/global-belonging

In‑Person Onboarding and Identity Verification

This role may require in-person onboarding and/or in‑person ID verification.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Full Stack Engineer, Applied AI
Senior Full Stack Engineer, Applied AI

Autodesk, Inc • Canada

Remote
CAD 123,000 - 180,000
Senior Full Stack Engineer, Applied AI
Senior Full Stack Engineer, Applied AI

Autodesk, Inc • Toronto

On-site
CAD 123,000 - 180,000
Senior AI Quality Developer, Personalization
Senior AI Quality Developer, Personalization

Autodesk, Inc • Toronto

On-site
CAD 84,000 - 123,000
AI Capability & Value Strategy Lead
AI Capability & Value Strategy Lead

Autodesk, Inc • Toronto

On-site
CAD 129,000 - 189,000
Director, Employee Relations
Director, Employee Relations

Autodesk, Inc • Canada

On-site
CAD 144,000 - 211,000
Data Engineer
Data Engineer

Autodesk, Inc • Toronto

On-site
CAD 88,000 - 129,000
Senior Full Stack Engineer, Applied AI
Senior Full Stack Engineer, Applied AI

Autodesk • Toronto

On-site
CAD 123,000 - 180,000
Senior Product Manager, Interoperability
Senior Product Manager, Interoperability

Autodesk, Inc • Toronto

On-site
CAD 122,000 - 179,000
Salary transparency
Global collaboration
Career growth
Principal Software Developer
Principal Software Developer

Autodesk, Inc • Montreal

On-site
CAD 131,000 - 191,000
Principal Software Developer
Principal Software Developer

Autodesk, Inc • Canada

On-site
CAD 131,000 - 191,000