Senior Data Engineer: AI-Driven Big Data Pipelines

Autodesk; Toronto; On-site; CAD 107,000 - 157,000; Full time

Job summary

Autodesk is seeking a Senior Data Engineer for our Compliance team to design, build, and optimize large-scale data pipelines and analytics. You will work with Python, SQL, Spark, Airflow, Hive, Snowflake, and AWS to enable scalable, compliant data processing and AI-assisted engineering workflows.

The role focuses on reliability, performance, and cost efficiency across batch and near real-time processing, with opportunities to impact product usage analytics and governance reporting.

Qualifications

• 4–7 years in data engineering or related roles.
• 3+ years with Spark, PySpark, Spark SQL.
• 3+ years Python and SQL with clean, modular code.
• Strong distributed computing fundamentals and Spark tuning.
• Hands-on Airflow experience.
• ETL/ELT design and large-scale processing knowledge.
• Experience with AI tools in development workflows.
• AWS data services and Snowflake/Hive/Presto familiarity.
• Git, CI/CD, and automation tooling.

Responsibilities

• Design and implement large-scale batch and near real-time data pipelines using Spark/PySpark.
• Build and optimize ETL/ELT workflows across distributed platforms.
• Integrate AI-assisted development practices into engineering workflows.
• Ensure data quality, observability, and reliable pipelines with monitoring and alerts.
• Collaborate with product managers, data scientists, and stakeholders to translate requirements into scalable solutions.
• Develop data models and curated datasets for compliance, analytics, and reporting.

Skills

Big data pipelines

Python

SQL

Apache Spark

Airflow

ETL/ELT design

AWS data services

Snowflake / Hive / Presto

Git / CI/CD

AI-assisted coding tools

Education

Bachelor's degree in CS/Engineering or related field

Tools

Spark / PySpark / Spark SQL

Apache Airflow

Hive, Snowflake, Presto/Trino

AWS (S3, EMR, Glue)

Git, Jenkins

Power BI / Looker

Job description

Autodesk is seeking a Senior Data Engineer for our Compliance team to design, build, and optimize large-scale data pipelines and analytics. You will work with Python, SQL, Spark, Airflow, Hive, Snowflake, and AWS to enable scalable, compliant data processing and AI-assisted engineering workflows.

The role focuses on reliability, performance, and cost efficiency across batch and near real-time processing, with opportunities to impact product usage analytics and governance reporting.