AWS Data Engineering

Compunnel Inc.; Toronto; On-site; CAD 90,000 - 120,000; Full time

Job summary

A leading tech company in Toronto is seeking an experienced AWS Data Engineer to design and implement scalable data pipelines and maintain cloud infrastructure. The ideal candidate will have expertise in AWS services like Glue, Redshift, and Lambda, along with proficiency in Python and SQL. This role involves collaboration with data teams to optimize data models and ensure compliance with data governance practices.

Qualifications

• Proficiency in AWS services such as Redshift, Glue, RDS, Lambda, and Airflow.
• Experience with implementing CI/CD pipelines for data engineering workflows.
• Strong understanding of data modeling, warehousing, and ETL processes.

Responsibilities

• Design and implement scalable ETL/ELT pipelines using AWS Glue and Lambda.
• Integrate data from diverse sources into data lakes and data warehouses.
• Monitor data pipelines and troubleshoot issues to ensure reliability.

Skills

Python

SQL

PySpark

AWS services (S3, Glue, Lambda, Redshift, RDS, Athena)

Orchestration tools (Apache Airflow, MWAA)

DevOps tools (CDK, CI/CD, Git, Terraform)

Tools

Terraform

CloudFormation

Data visualization tools (e.g., Power BI)

Job description

We are seeking a skilled AWS Data Engineer with expertise in building scalable data pipelines and cloud-native platforms.

The ideal candidate will have hands-on experience with AWS services such as Redshift, Glue, RDS, Lambda, and Airflow, and will be responsible for designing, implementing, and maintaining robust data engineering workflows that support analytics, reporting, and machine learning initiatives.

Key Responsibilities

• Design and implement scalable ETL/ELT pipelines using AWS Glue, Lambda, Step Functions, and related services.
• Integrate structured and unstructured data from diverse sources into data lakes and warehouses (e.g., S3, Redshift, RDS, Athena).
• Build and maintain cloud infrastructure for data analytics platforms using Terraform, CloudFormation, or similar IaC tools.
• Develop reusable frameworks and automation scripts to streamline data operations.

Data Analytics Enablement

• Collaborate with data engineers, data scientists, and analysts to deliver high-quality platforms for data loading, reporting, and machine learning.
• Optimize data models and queries for performance and scalability.

Monitoring and Maintenance

• Monitor data pipelines and troubleshoot issues to ensure reliability and data integrity.
• Implement CI/CD pipelines for data engineering workflows using GitLab, Bitbucket, Jenkins, or GitHub Actions.

Security and Governance

• Ensure compliance with data governance and security best practices.
• Implement access controls and encryption for sensitive data.

Required Qualifications

• Proficiency in Python, SQL, and PySpark.
• Hands-on experience with AWS services: S3, Glue, Lambda, Redshift, RDS, Athena, Step Functions.
• Experience with orchestration tools like Apache Airflow or MWAA.
• Familiarity with DevOps tools and practices (CDK, CI/CD, Git, Terraform).
• Strong understanding of data modeling, warehousing, and ETL processes.

Preferred Qualifications

• Experience with big data tools (Spark, Hive, Kafka, Python, TypeScript).
• Knowledge of containerization (Docker, Kubernetes).
• Familiarity with data visualization tools (e.g., Power BI).
• AWS certifications (e.g., AWS Certified Data Analytics – Specialty).