Principal Data Engineer (PySpark, AI)

Autodesk; Bengaluru; On-site; INR 3,000,000 - 5,500,000; Full time

Job summary

Autodesk is seeking a Principal Data Engineer to join the Advanced Compliance Products team, leading scalable data platforms that process billions of records in a cost-efficient manner.

The role focuses on building and optimizing large-scale data pipelines using Spark, PySpark, Python, and SQL, while guiding architecture and mentoring engineers across teams. Collaboration with product managers and data scientists will translate business needs into robust data lakehouse solutions.

Qualifications

• 10+ years of experience in data engineering, distributed systems, or large-scale data processing.
• Proven expertise in Apache Spark, PySpark, Python, Spark SQL, and SQL, with tuning/optimisation.
• Experience architecting and scaling complex data pipelines and distributed processing systems.
• Strong knowledge of AWS, Hive, Snowflake, and modern data lakehouse architectures.
• Experience with ETL/ELT, orchestration tools like Airflow, and CI/CD practices.
• Hands-on experience with AI/GenAI technologies and tooling (LLMs, prompts, Copilot, etc.).
• Experience applying AI to data engineering workflows (code generation, data quality, anomaly detection, etc.).
• Strong understanding of data quality, observability, reliability, and cloud infrastructure.
• Demonstrated technical leadership, architectural decision-making, and mentoring experience.

Responsibilities

• Define and drive the architecture of scalable, reliable, and cost-efficient data platforms processing billions of records.
• Lead the design, development, and optimization of large-scale data pipelines using Spark, PySpark, Python, and SQL.
• Drive platform modernization initiatives, including incremental processing, reusable frameworks, and modern data lakehouse architectures.
• Leverage AI/GenAI technologies to improve engineering productivity, automate workflows, enhance data quality, and optimize pipeline operations.
• Identify and resolve complex performance, scalability, reliability, and data quality challenges.
• Establish engineering best practices for CI/CD, automated testing, observability, and cloud infrastructure.
• Lead technical design discussions, influence architectural decisions, and mentor engineers across teams.
• Collaborate with product managers, data scientists, and business stakeholders to translate requirements into scalable data solutions.
• Take end-to-end technical ownership of cross-team engineering initiatives.

Skills

Spark

PySpark

Python

SQL

Airflow

Iceberg

LLMs

GenAI

Education

Bachelor's degree in CS/Engineering

Tools

AWS

Job description

Position Overview

Autodesk is looking for a highly skilled Principal Data Engineer to join our Advanced Compliance Products (ACP) team. ACP is a strategic initiative focused on detecting non-compliant usage of Autodesk software through large-scale data processing and analytics.

Job Requisition ID #

26WD101628

Responsibilities

• Define and drive the architecture of scalable, reliable, and cost-efficient data platforms processing billions of records.
• Lead the design, development, and optimization of large-scale data pipelines using Spark, PySpark, Python, and SQL.
• Drive platform modernization initiatives, including incremental processing, reusable frameworks, and modern data lakehouse architectures.
• Leverage AI/GenAI technologies and AI-assisted development tools to improve engineering productivity, automate workflows, enhance data quality, and optimize pipeline operations.
• Identify and resolve complex performance, scalability, reliability, and data quality challenges.
• Establish engineering best practices for CI/CD, automated testing, observability, and cloud infrastructure.
• Lead technical design discussions, influence architectural decisions, and mentor engineers across teams.
• Collaborate with product managers, data scientists, and business stakeholders to translate business requirements into scalable data solutions.
• Take end-to-end technical ownership of complex, cross-team engineering initiatives.

Minimum Qualifications

• 10+ years of experience in data engineering, distributed systems, or large-scale data processing.
• Strong hands-on expertise in Apache Spark, PySpark, Python, Spark SQL, and SQL, including performance tuning and optimization.
• Proven experience architecting and scaling complex data pipelines and distributed data processing systems.
• Strong knowledge of AWS, Hive, Snowflake, and modern data lakehouse architectures.
• Experience with data modeling, ETL/ELT frameworks, orchestration tools (such as Airflow), and CI/CD practices.
• Hands-on experience with AI/GenAI technologies, including LLMs, prompt engineering, and AI-assisted development tools such as GitHub Copilot, Cursor, or equivalent.
• Practical experience applying AI to data engineering workflows, such as code generation, automated testing, data quality validation, anomaly detection, pipeline optimization, or intelligent troubleshooting.
• Strong understanding of data quality, observability, reliability, and cloud infrastructure.
• Demonstrated technical leadership, architectural decision-making, and mentoring experience.
• Excellent problem-solving, communication, and cross-functional collaboration skills.
• Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.

Preferred Qualifications

• Experience with Apache Iceberg, incremental data processing, and modern orchestration frameworks.
• Experience modernizing legacy data platforms and optimizing large-scale Spark workloads.
• Familiarity with BI tools such as Power BI or Looker.
• Experience with AI agents or LLM-powered automation.
• Experience building reusable data engineering frameworks and driving engineering standards across teams.

About Autodesk / À Propos D’Autodesk

Welcome to Autodesk! Amazing things are created every day with our software – from the greenest buildings and cleanest cars to the smartest factories and biggest hit movies. We help innovators turn their ideas into reality, transforming not only how things are made, but what can be made.

We take great pride in our culture here at Autodesk – it’s at the core of everything we do. Our culture guides the way we work and treat each other, informs how we connect with customers and partners, and defines how we show up in the world.

When you’re an Autodesker, you can do meaningful work that helps build a better world designed and made for all. Ready to shape the world and your future? Join us!

Bienvenue chez Autodesk ! Nos logiciels créent chaque jour des choses extraordinaires : des bâtiments les plus écologiques aux voitures les plus propres, en passant par les usines les plus intelligentes et les films à succès. Nous aidons les innovateurs à concrétiser leurs idées, transformant non seulement la façon dont les choses sont fabriquées, mais aussi ce qui peut l’être.

Nous sommes très fiers de notre culture chez Autodesk ; elle est au cœur de tout ce que nous faisons. Notre culture guide notre façon de travailler et de nous comporter les uns envers les autres, influence nos interactions avec nos clients et nos partenaires, et définit notre image au monde.

En tant qu’Autodeskien, vous pouvez accomplir un travail significatif qui contribue à bâtir un monde meilleur, conçu et réalisé pour tous. Prêt à façonner le monde et votre avenir ? Rejoignez-nous !

Salary transparency / Transparence Salariale

Salary is one part of Autodesk’s competitive compensation package. Offers are based on the candidate’s experience and geographic location. In addition to base salaries, our compensation package may include annual cash bonuses, commissions for sales roles, stock grants, and a comprehensive benefits package.

Belonging

We take pride in cultivating a culture of belonging where everyone can thrive. Learn more here: https://www.autodesk.com/company/global-belonging

In-Person Onboarding and Identity Verification

This role may require in-person onboarding and/or in-person ID verification.