Senior Data Engineer: AI-Driven Big Data Pipelines

Autodesk, Inc.

Toronto

On-site

CAD 107,000 - 157,000

Full time

2 days ago

Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Autodesk, Inc. is seeking a Senior Data Engineer to join our Compliance team. You will design, build, and optimize scalable data pipelines handling billions of records using Apache Spark, PySpark, and Spark SQL.

The role emphasizes AI-assisted development practices to accelerate engineering workflows, improve code quality, and deliver solutions faster. You will work with Hive, Snowflake, AWS S3, EMR, and Glue, building robust ETL/ELT pipelines and production-grade data workflows with Airflow.

Qualifications

4–7 years of relevant experience in data engineering, big data systems, or software engineering.
3+ years of hands-on experience with Apache Spark, including PySpark, Spark SQL, and DataFrame APIs.
3+ years of strong programming experience in Python and SQL, writing clean, modular, maintainable, and testable code.
Strong understanding of distributed computing fundamentals, including parallel processing, partitioning, shuffling, data skew, fault tolerance, and Spark performance tuning.
Hands-on experience with Apache Airflow for building, scheduling, and orchestrating complex data pipelines.

Responsibilities

Leverage AI-powered development tools to accelerate coding, debugging, refactoring, test generation, documentation, and troubleshooting of complex data engineering solutions
Integrate AI-assisted development practices into engineering workflows to improve productivity, code quality, and delivery speed
Evaluate and validate AI-generated solutions for correctness, performance, maintainability, and security while following responsible AI practices
Identify opportunities to automate repetitive engineering tasks and improve developer efficiency through AI-powered solutions
Design, develop, and optimize large-scale batch and near-real-time data processing pipelines using Apache Spark, PySpark, and Spark SQL
Build scalable and efficient ETL/ELT pipelines capable of processing billions of records across distributed data platforms
Optimize distributed processing workloads through partitioning, efficient joins, query tuning, resource management, and performance analysis
Develop and maintain data processing solutions using Hive, Snowflake, AWS S3, EMR, and Glue
Build and maintain production-grade data workflows using Apache Airflow, including DAG development, scheduling, dependency management, retries, and failure recovery
Implement data quality checks, pipeline monitoring, observability, and alerting to ensure reliable data processing
Design reusable, modular, and fault-tolerant pipelines supporting incremental processing, backfills, and operational scalability
Troubleshoot production issues and continuously improve pipeline reliability, performance, and cost efficiency
Develop and maintain scalable, cloud-native data infrastructure on AWS
Automate infrastructure provisioning, deployments, and operational processes using Infrastructure as Code and CI/CD practices
Implement automated testing, version control, code reviews, and software engineering best practices
Own the end-to-end delivery of data engineering solutions, from technical design and implementation to deployment and production support
Partner with product managers, data scientists, analysts, and business stakeholders to translate business requirements into scalable technical solutions
Design efficient data models and curated datasets to support compliance detection, product usage analytics, and business reporting
Define and promote data engineering standards, reusable frameworks, and architectural best practices
Communicate technical designs, trade-offs, and engineering outcomes effectively to stakeholders