Principal Data Engineer (PySpark, AI)

Autodesk, Inc.; Bengaluru; Hybrid; INR 4,000,000 - 6,500,000; Full time

Job summary

Autodesk, Inc. is seeking a Principal Data Engineer to lead the ACP data platform, shaping architecture for scalable pipelines, modern data lakehouse, and AI-assisted workflows.

You will own end-to-end design, collaborate with product managers and data scientists, and mentor engineers across teams in Bengaluru and beyond. You will drive modernization using Spark, PySpark, Python, and SQL while ensuring data quality, observability, and robust CI/CD practices.

Qualifications

• 10+ years of experience in data engineering, distributed systems, or large-scale data processing.
• Strong hands-on expertise in Apache Spark, PySpark, Python, Spark SQL, and SQL, including performance tuning and optimization.
• Proven experience architecting and scaling complex data pipelines and distributed data processing systems.

Responsibilities

• Define and drive the architecture of scalable, reliable data platforms processing billions of records.
• Lead the design, development, and optimization of large-scale data pipelines using Spark, PySpark, Python, and SQL.
• Drive platform modernization initiatives, including incremental processing, reusable frameworks, and modern data lakehouse architectures.
• Leverage AI/GenAI technologies and AI-assisted development tools to improve engineering productivity, automate workflows, enhance data quality, and optimize pipeline operations.
• Identify and resolve performance, scalability, reliability, and data quality challenges.
• Establish engineering best practices for CI/CD, automated testing, observability, and cloud infrastructure.
• Lead technical design discussions, influence architectural decisions, and mentor engineers across teams.
• Collaborate with product managers, data scientists, and business stakeholders to translate business requirements into scalable data solutions.
• Take end-to-end technical ownership of complex, cross-team engineering initiatives.

Skills

Spark

PySpark

Python

SQL

AWS

Airflow

Iceberg

Snowflake

Education

B.E./B.Tech in CS or related

Tools

Apache Iceberg

Snowflake

Airflow

CI/CD

Job description

Job Requisition ID # 26WD101628 Position Overview Autodesk is looking for a highly skilled Principal Data Engineer to join our Advanced Compliance Products (ACP) team. ACP is a strategic initiative focused on detecting non-compliant usage of Autodesk software through large-scale data processing and analytics. As a Principal Data Engineer, you will drive the technical architecture, scalability, and modernization of ACP's data engineering platform, leveraging modern cloud technologies and AI-driven engineering practices. You will solve complex engineering challenges, establish best practices, and lead cross-functional technical initiatives while remaining hands-on.

Responsibilities

• Define and drive the architecture of scalable, reliable, and cost-efficient data platforms processing billions of records.
• Lead the design, development, and optimization of large-scale data pipelines using Spark, PySpark, Python, and SQL.
• Drive platform modernization initiatives, including incremental processing, reusable frameworks, and modern data lakehouse architectures.
• Leverage AI/GenAI technologies and AI-assisted development tools to improve engineering productivity, automate workflows, enhance data quality, and optimize pipeline operations.
• Identify and resolve complex performance, scalability, reliability, and data quality challenges.
• Establish engineering best practices for CI/CD, automated testing, observability, and cloud infrastructure.
• Lead technical design discussions, influence architectural decisions, and mentor engineers across teams.
• Collaborate with product managers, data scientists, and business stakeholders to translate business requirements into scalable data solutions.
• Take end-to-end technical ownership of complex, cross-team engineering initiatives.

Minimum Qualifications

• 10+ years of experience in data engineering, distributed systems, or large-scale data processing.
• Strong hands-on expertise in Apache Spark, PySpark, Python, Spark SQL, and SQL, including performance tuning and optimization.
• Proven experience architecting and scaling complex data pipelines and distributed data processing systems.
• Strong knowledge of AWS, Hive, Snowflake, and modern data lakehouse architectures.
• Experience with data modeling, ETL/ELT frameworks, orchestration tools (such as Airflow), and CI/CD practices.
• Hands-on experience with AI/GenAI technologies, including LLMs, prompt engineering, and AI-assisted development tools such as GitHub Copilot, Cursor, or equivalent.
• Practical experience applying AI to data engineering workflows, such as code generation, automated testing, data quality validation, anomaly detection, pipeline optimization, or intelligent troubleshooting.
• Strong understanding of data quality, observability, reliability, and cloud infrastructure.
• Demonstrated technical leadership, architectural decision-making, and mentoring experience.
• Excellent problem-solving, communication, and cross-functional collaboration skills.
• Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.

Preferred Qualifications

• Experience with Apache Iceberg, incremental data processing, and modern orchestration frameworks.
• Experience modernizing legacy data platforms and optimizing large-scale Spark workloads.
• Familiarity with BI tools such as Power BI or Looker.
• Experience with AI agents or LLM-powered automation.
• Experience building reusable data engineering frameworks and driving engineering standards across teams.

Welcome to Autodesk! Amazing things are created every day with our software – from the greenest buildings and cleanest cars to the smartest factories and biggest hit movies. We help innovators turn their ideas into reality, transforming not only how things are made, but what can be made. We take great pride in our culture here at Autodesk – it’s at the core of everything we do. Our culture guides the way we work and treat each other, informs how we connect with customers and partners, and defines how we show up in the world. When you’re an Autodesker, you can do meaningful work that helps build a better world designed and made for all. Ready to shape the world and your future? Join us! Bienvenue chez Autodesk ! Nos logiciels créent chaque jour des choses extraordinaires : des bâtiments les plus écologiques aux voitures les plus propres, en passant par les usines les plus intelligentes et les films à succès. Nous aidons les innovateurs à concrétiser leurs idées, transformant non seulement la façon dont les choses sont fabriquées, mais aussi ce qui peut l’être. Nous sommes très fiers de notre culture chez Autodesk ; elle est au cœur de tout ce que nous faisons. Notre culture guide notre façon de travailler et de nous comporter les uns envers les autres, influence nos interactions avec nos clients et nos partenaires, et définit notre image au monde. En tant qu’Autodeskien, vous pouvez accomplir un travail significatif qui contribue à bâtir un monde meilleur, conçu et réalisé pour tous. Prêt à façonner le monde et votre avenir ? Rejoignez-nous !

Salary transparency / Transparence Salariale

Salary is one part of Autodesk’s competitive compensation package. Offers are based on the candidate’s experience and geographic location. In addition to base salaries, our compensation package may include annual cash bonuses, commissions for sales roles, stock grants, and a comprehensive benefits package.

Belonging

We take pride in cultivating a culture of belonging where everyone can thrive. Learn more here: https://www.autodesk.com/company/global-belonging

In-Person Onboarding and Identity Verification

This role may require in-person onboarding and/or in-person ID verification.

Welcome to Autodesk! Amazing things are created every day with our software – from the greenest buildings and cleanest cars to the smartest factories and biggest hit movies. We help innovators turn their ideas into reality, transforming not only how things are made, but what can be made. We take great pride in our culture here at Autodesk – it’s at the core of everything we do. Our culture guides the way we work and treat each other, informs how we connect with customers and partners, and defines how we show up in the world. When you’re an Autodesker, you can do meaningful work that helps build a better world designed and made for all. Ready to shape the world and your future? Join us! Bienvenue chez Autodesk ! Nos logiciels créent chaque jour des choses extraordinaires : des bâtiments les plus écologiques aux voitures les plus propres, en passant par les usines les plus intelligentes et les films à succès. Nous aidons les innovateurs à concrétiser leurs idées, transformant non seulement la façon dont les choses sont fabriquées, mais aussi ce qui peut l’être. Nous sommes très fiers de notre culture chez Autodesk ; elle est au cœur de tout ce que nous faisons. Notre culture guide notre façon de travailler et de nous comporter les uns envers les autres, influence nos interactions avec nos clients et nos partenaires, et définit notre image au monde. En tant qu’Autodeskien, vous pouvez accomplir un travail significatif qui contribue à bâtir un monde meilleur, conçu et réalisé pour tous. Prêt à façonner le monde et votre avenir ? Rejoignez-nous !