Principal Data Engineer

Autodesk India Pvt Ltd.; Bengaluru; On-site; INR 4,200,000 - 6,200,000; Full time

Job summary

Autodesk India Pvt Ltd. is seeking a Senior Principal Data Engineer to architect and scale data platforms for Advanced Compliance Products (ACP). The role focuses on Spark, PySpark, Python, and SQL on cloud architectures to enable AI-driven data processing and governance at scale.

You will lead cross-functional initiatives, mentor engineers, and collaborate with data scientists and product managers to translate requirements into robust data solutions.

Qualifications

• 10+ years of experience in data engineering or distributed data processing.
• Strong hands-on expertise in Spark, PySpark, Python and SQL with performance tuning.
• Experience with AWS, Hive, Snowflake, and data lakehouse architectures.
• Experience with AI/GenAI tools in data engineering workflows.

Responsibilities

• Define and drive scalable, cost-efficient data platforms handling billions of records.
• Lead design and optimization of large-scale data pipelines using Spark, PySpark, Python, and SQL.
• Drive platform modernization with incremental processing and modern data lakehouse architectures.
• Leverage AI/GenAI to improve productivity, data quality, and pipeline operations.
• Mentor engineers and lead cross-team technical discussions.

Skills

Spark

PySpark

Python

SQL

Data pipelines

AWS

Data modeling

Education

Bachelor's degree in CS/Engineering

Tools

AWS

Hive

Snowflake

Airflow

CI/CD

GitHub Copilot

Cursor

Job description

Position Overview

Principal Data Engineer (PySpark, AI) Job Requisition ID # 26WD101628 Autodesk is looking for a highly skilled Principal Data Engineer to join our Advanced Compliance Products (ACP) team. ACP is a strategic initiative focused on detecting non-compliant usage of Autodesk software through large-scale data processing and analytics. As a Principal Data Engineer, you will drive the technical architecture, scalability, and modernization of ACP's data engineering platform, leveraging modern cloud technologies and AI-driven engineering practices. You will solve complex engineering challenges, establish best practices, and lead cross-functional technical initiatives while remaining hands‑on.

Responsibilities

• Define and drive the architecture of scalable, reliable, and cost-efficient data platforms processing billions of records.
• Lead the design, development, and optimization of large-scale data pipelines using Spark, PySpark, Python, and SQL.
• Drive platform modernization initiatives, including incremental processing, reusable frameworks, and modern data lakehouse architectures.
• Leverage AI/GenAI technologies and AI-assisted development tools to improve engineering productivity, automate workflows, enhance data quality, and optimize pipeline operations.
• Identify and resolve complex performance, scalability, reliability, and data quality challenges.
• Establish engineering best practices for CI/CD, automated testing, observability, and cloud infrastructure.
• Lead technical design discussions, influence architectural decisions, and mentor engineers across teams.
• Collaborate with product managers, data scientists, and business stakeholders to translate business requirements into scalable data solutions.
• Take end‑to‑end technical ownership of complex, cross‑team engineering initiatives.

Minimum Qualifications

• 10+ years of experience in data engineering, distributed systems, or large-scale data processing.
• Strong hands‑on expertise in Apache Spark, PySpark, Python, Spark SQL, and SQL, including performance tuning and optimization.
• Proven experience architecting and scaling complex data pipelines and distributed data processing systems.
• Strong knowledge of AWS, Hive, Snowflake, and modern data lakehouse architectures.
• Experience with data modeling, ETL/ELT frameworks, orchestration tools (such as Airflow), and CI/CD practices.
• Hands‑on experience with AI/GenAI technologies, including LLMs, prompt engineering, and AI-assisted development tools such as GitHub Copilot, Cursor, or equivalent.
• Practical experience applying AI to data engineering workflows, such as code generation, automated testing, data quality validation, anomaly detection, pipeline optimization, or intelligent troubleshooting.
• Strong understanding of data quality, observability, reliability, and cloud infrastructure.
• Demonstrated technical leadership, architectural decision‑making, and mentoring experience.
• Excellent problem‑solving, communication, and cross‑functional collaboration skills.
• Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.

Preferred Qualifications

• Experience with Apache Iceberg, incremental data processing, and modern orchestration frameworks.
• Experience modernizing legacy data platforms and optimizing large-scale Spark workloads.
• Familiarity with BI tools such as Power BI or Looker.
• Experience with AI agents or LLM‑powered automation.
• Experience building reusable data engineering frameworks and driving engineering standards across teams.

About Autodesk

Welcome to Autodesk! Amazing things are created every day with our software – from the greenest buildings and cleanest cars to the smartest factories and biggest hit movies. We help innovators turn their ideas into reality, transforming not only how things are made, but what can be made. We take great pride in our culture here at Autodesk – it’s at the core of everything we do. Our culture guides the way we work and treat each other, informs how we connect with customers and partners, and defines how we show up in the world. When you’re an Autodesker, you can do meaningful work that helps build a better world designed and made for all. Ready to shape the world and your future? Join us! Bienvenue chez Autodesk ! Nos logiciels créent chaque jour des choses extraordinaires : des bâtiments les plus écologiques aux voitures les plus propres, en passant par les usines les plus intelligentes et les films à succès. Nous aidons les innovateurs à concrétiser leurs idées, transformant non seulement la façon dont les choses sont fabriquées, mais aussi ce qui peut l’être. Nous sommes très fiers de notre culture chez Autodesk ; elle est au cœur de tout ce que nous faisons. Notre culture guide notre façon de travailler et de nous comporter les uns envers les autres, influence nos interactions avec nos clients et nos partenaires, et définit notre image au monde. En tant qu’Autodeskien, vous pouvez accomplir un travail significatif qui contribue à bâtir un monde meilleur, conçu et réalisé pour tous. Prêt à façonner le monde et votre avenir ? Rejoignez-nous !

Salary transparency

Salary is one part of Autodesk’s competitive compensation package. Offers are based on the candidate’s experience and geographic location. In addition to base salaries, our compensation package may include annual cash bonuses, commissions for sales roles, stock grants, and a comprehensive benefits package.

Belonging

We take pride in cultivating a culture of belonging where everyone can thrive. Learn more here: https://www.autodesk.com/company/global-belonging

In-Person Onboarding and Identity Verification

This role may require in-person onboarding and/or in-person ID verification.

Experience Level

Senior Level