Senior Generative AI Engineer

Autodesk India Pvt Ltd.; Bengaluru; On-site; INR 4,500,000 - 6,500,000; Full time

Job summary

Autodesk India Pvt Ltd. in Bengaluru seeks a Senior AI Engineer focused on LLM evaluation to build and scale AI quality systems. You will establish evaluation standards, develop automated validation, and drive observability for production AI demos.

You will collaborate with product managers, data scientists, and engineers to deploy AI-powered features, implement testing pipelines, and ensure responsible AI practices across cloud-native systems.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, ML, Data Science, Engineering, or equivalent.
• 5+ years of hands-on experience delivering AI/ML-powered systems into production.
• Experience designing evaluation frameworks, automated testing solutions, or model validation pipelines.
• Strong understanding of LLM evaluation, AI observability, and quality metrics.

Responsibilities

• Lead AI evaluation and quality strategy for Generative AI and ML-powered systems.
• Design scalable evaluation frameworks for LLMs, RAG systems, and agentic workflows.
• Fine-tune, evaluate, benchmark, and deploy large language models in production.
• Implement evaluation and observability standards including regression testing and monitoring.
• Define and track AI quality metrics such as accuracy, bias, drift, latency, and hallucination rates.
• Build automated evaluation pipelines and integrate them into CI/CD workflows.
• Collaborate with product managers, architects, and data scientists to deliver AI-powered experiences.

Skills

Python

AI/ML engineering

CI/CD

Technical communication

Education

Bachelor’s or Master’s degree in Computer Science, ML/DS/Engineering

Tools

LangSmith

DeepEval

Prompt testing tools

Job description

Senior AI Engineer - LLM Evaluation

Job Requisition ID # 26WD98447

Position Overview

If you love building real AI systems that real customers use and you are passionate about measuring, improving, and scaling AI quality this role is for you. The Applied AI team in Autodesk's AEC organization builds cloud-native AI solutions that make Architecture, Engineering, and Construction workflows smarter, faster, and more connected. This role is primarily focused on AI evaluation and quality engineering for Generative AI systems. You will establish evaluation standards, build automated validation frameworks, and drive observability practices that ensure AI systems are reliable, trustworthy, and production-ready. While evaluation is the primary focus, you will also contribute to the design and development of AI-powered applications including RAG systems, agentic workflows, and LLM-based solutions deployed at scale.

Responsibilities

• Lead AI evaluation and quality strategy for Generative AI and ML-powered systems
• Design and implement scalable evaluation frameworks for LLMs, RAG systems, agentic workflows, and traditional ML models
• Fine-tune, evaluate, benchmark, and deploy large language models in production environments
• Implement evaluation and observability standards including regression testing, monitoring, tracing, feedback loops, and quality measurement for AI systems
• Define and track AI quality metrics including accuracy, robustness, bias, drift, grounding, latency, safety, and hallucination rates
• Build automated evaluation pipelines and integrate them into CI/CD workflows for continuous model validation
• Evaluate and adopt AI testing frameworks, benchmark methodologies, and emerging evaluation tooling
• Build scalable RAG pipelines using structured and unstructured enterprise data sources
• Integrate AI agents with Autodesk platforms and enterprise services using MCP-based integrations or equivalent patterns
• Partner with product managers, architects, developers, designers, and data scientists to deliver AI-powered experiences
• Ensure AI systems align with Responsible AI principles and enterprise governance requirements
• Contribute to reusable AI platform capabilities, SDKs, orchestration patterns, and engineering best practices
• Use AI coding agents to accelerate feature delivery while ensuring rigorous validation through tests, CI/CD, code reviews, and security practices

Minimum Qualifications

• Bachelor’s or Master’s degree in Computer Science, Machine Learning, Data Science, Engineering, or equivalent practical experience
• 5+ years of hands‑on experience delivering AI/ML-powered systems into production
• Experience designing evaluation frameworks, automated testing solutions, benchmarking systems, or model validation pipelines
• Strong understanding of LLM evaluation, model quality measurement, and AI observability concepts
• Strong proficiency in Python with experience building automation frameworks and testing infrastructure
• Experience deploying and operating AI applications in production, including monitoring, observability, and continuous improvement
• Experience building cloud-native applications on AWS, Azure, or Google Cloud
• Excellent technical communication and collaboration skills
• Demonstrated experience using AI coding tools to accelerate delivery of production systems

Preferred Qualifications

• Hands‑on experience with AI evaluation frameworks such as LangSmith, DeepEval, TruLens, Promptfoo, OpenAI Evals, or similar tools
• Experience evaluating RAG systems, agentic workflows, AI assistants, and prompt-based applications
• Familiarity with Responsible AI, AI governance, model risk assessment, and AI observability platforms
• Experience with cloud-native AI deployment using AWS, Azure, or Google Cloud
• Familiarity with deep learning architectures such as Transformers and modern ML frameworks including PyTorch, Lightning, Ray, or equivalent technologies
• Experience with enterprise knowledge systems, semantic search, vector search, or contextual retrieval systems
• Experience with emerging Agentic AI frameworks such as LangGraph, CrewAI, OpenAI SDKs, Claude SDKs, MCP, A2A, or equivalent technologies
• Contributions to open-source AI projects, technical blogs, conference talks, or publications are a plus

Salary transparency

Salary is one part of Autodesk's competitive compensation package. Offers are based on the candidate's experience and geographic location. In addition to base salaries, our compensation package may include annual cash bonuses, commissions for sales roles, stock grants, and a comprehensive benefits package.

Belonging

We take pride in cultivating a culture of belonging where everyone can thrive. Learn more here: https://www.autodesk.com/company/global-belonging

Experience Level

Senior Level