Senior Data Engineer — AWS Data Platform & Pipelines

Luxoft; Toronto; On-site; CAD 90,000 - 130,000; Full time

Job summary

Luxoft seeks an experienced Senior Data Engineer to onboard, integrate, and operationalize datasets for Middle Office functions. You will build scalable, production-grade data pipelines on AWS (EMR, Glue, Athena, S3) using Spark/PySpark, and own the full lifecycle from design to deployment and support.

Responsibilities include data onboarding, pipeline development, orchestration with Airflow, data modeling, and IaC with Terraform.

Qualifications

• Strong hands-on experience building and operating AWS-based data platforms and pipelines.
• Advanced expertise with Apache Spark, PySpark, distributed data processing, and performance optimization.
• Experience with modern data lake technologies, particularly Apache Hudi and Apache Iceberg.
• Strong Python programming and software engineering practices, including testing, code quality, and Git-based development workflows.
• Hands-on experience with Apache Airflow or comparable workflow orchestration platforms.
• Experience with Terraform, CI/CD pipelines, and infrastructure automation.
• Ability to independently translate business requirements into production-ready data solutions.

Responsibilities

• Data Onboarding & Integration: onboard new datasets and design scalable ingestion and transformation pipelines.
• Data Engineering: develop and maintain production-grade pipelines using AWS services (EMR, Glue, Athena, S3) with Spark/PySpark.
• Workflow Automation: build orchestration workflows using Airflow or equivalent, including scheduling and monitoring.
• Data Architecture & Modeling: design scalable data models, partitioning strategies, and data quality controls.
• DevOps & Operational Excellence: implement CI/CD pipelines and IaC using Terraform and GitHub Actions.
• Production Support: troubleshoot data quality, performance, and operational issues to improve reliability.

Skills

AWS data pipelines

Spark / PySpark

Python programming

Airflow

CI/CD

Terraform

Data modeling

Data quality

Production deployments

Tools

AWS

EMR

Glue

Athena

S3

Apache Hudi

Iceberg

Airflow

GitHub Actions

Job description

Luxoft seeks an experienced Senior Data Engineer to onboard, integrate, and operationalize datasets for Middle Office functions. You will build scalable, production-grade data pipelines on AWS (EMR, Glue, Athena, S3) using Spark/PySpark, and own the full lifecycle from design to deployment and support.

Responsibilities include data onboarding, pipeline development, orchestration with Airflow, data modeling, and IaC with Terraform.