Applied AI Developer (Agent Evaluation)

Autodesk; Toronto; On-site; CAD 110,000 - 160,000; Full time

Job summary

Autodesk in Toronto is seeking a Software Developer for the Fusion platform services team to help build cutting-edge AI and generative AI solutions that boost developer productivity and experience.

You will collaborate with AI engineers and product teams to create intelligent agentic systems, evaluate them, and integrate MCP-based tooling with IDEs such as VS Code. This role emphasizes hands-on development and rigorous evaluation.

Qualifications

• BS/MS in Computer Science, ML, or related applied AI field.
• Proficiency in Python and ML frameworks (PyTorch, Transformers, scikit-learn).
• Experience with LLMs applied to software understanding or test generation.
• Knowledge of AI evaluation methodologies and metrics for agentic tasks.
• Strong foundation in statistical analysis and experimental design.
• Experience with developer workflows and productivity measurement frameworks.

Responsibilities

• Develop and orchestrate multi-agent AI systems for automated test generation, test execution, and workflow optimization using LangGraph, AutoGen, or Claude Code.
• Design agentic workflows coordinating multiple AI agents to drive test automation across UI, API, integration, and system levels, integrating with MCP-compatible services.
• Build evaluation frameworks and benchmarks for agentic systems, including comparisons of AI agents against commercial solvers using AgentBench and Langfuse.
• Evaluate MCP server and tool performance across agentic pipelines, measuring latency, accuracy, context fidelity, and end-to-end task completion rates.

Skills

Python

ML frameworks

Large Language Models

AI evaluation metrics

Statistical analysis

Experiment design

CI/CD

Education

BS/MS in Computer Science or ML

Tools

LangGraph

AutoGen

Anthropic Agent SDK

Langfuse

MCP

VS Code

Job description

Position Overview

As a Software Developer on the Fusion platform services team within Product Development and Manufacturing Solutions (PDMS), you'll be part of a team of technologists dedicated to creating cutting-edge AI and generative AI solutions that enhance developer productivity and experience. You'll work closely with AI engineers, software architects, and product engineering teams to build and rigorously evaluate intelligent agentic systems — including benchmarking AI agents against commercial solvers — and develop MCP (Model Context Protocol)-based tooling that integrates seamlessly with IDEs such as VS Code and Cursor.

Responsibilities

• Develop and orchestrate multi-agent AI systems for automated test generation, test execution, and end-to-end development workflow optimization using frameworks like LangGraph, AutoGen, or the Anthropic Agent SDK (Claude Code)
• Design and implement agentic workflows that coordinate multiple AI agents to autonomously drive test automation across UI, API, integration, and system levels, from test case synthesis to result evaluation, ensuring seamless integration with existing developer tools and MCP-compatible services
• Build evaluation frameworks and custom benchmarks for agentic systems, including comparisons of AI agents against commercial solvers, using tools like AgentBench and Langfuse
• Evaluate MCP server and tool performance across agentic pipelines, measuring latency, accuracy, context fidelity, and end-to-end task completion rates

Minimum Qualifications

• BS/MS in Computer Science, Machine Learning, or a related applied AI field
• Expertise in Python and ML frameworks (PyTorch, Transformers, scikit-learn)
• Experience with Large Language Models applied to software understanding or test generation
• Knowledge of AI evaluation methodologies and metrics for agentic task completion and test quality
• Strong foundation in statistical analysis and experimental design
• Experience with developer workflow and productivity measurement frameworks

Preferred Qualifications

• Background in software engineering or QA with close collaboration with development teams
• Familiarity with test automation frameworks (e.g., Playwright, Selenium, Pytest, Appium) and CI/CD pipelines
• Experience designing benchmarks that compare AI agents against commercial or domain-specific solvers
• Hands-on experience with MCP (Model Context Protocol), building, evaluating, and optimizing MCP servers and tool integrations within agentic pipelines
• Experience with agentic AI frameworks including LangGraph, AutoGen, or the Anthropic Agent SDK / Claude Code
• Knowledge in vision-language models or multi-modal AI for UI and system-level understanding and evaluation
• Experience with Azure AI Foundry/ML or AWS cloud ML platforms