Data Engineer Python/PySpark/OpenShift

Compunnel Inc.; Mississauga; On-site; CAD 100,000 - 130,000; Full time

Job summary

A leading data engineering firm is seeking a Data Engineer to design and maintain scalable data pipelines using Python and PySpark. This role requires managing data storage solutions, orchestrating ETL processes, and deploying containerized applications. The ideal candidate will have extensive experience in SQL, data orchestration tools, and cloud platforms. Strong communication and teamwork abilities are essential, along with a solid understanding of data security and governance.

Qualifications

• Minimum 10 years of overall experience in data engineering or similar roles.
• Expertise in SQL and experience with Snowflake Data Warehouse.
• Strong proficiency in Python and PySpark.

Responsibilities

• Develop, construct, test, and maintain scalable data pipelines.
• Manage and optimize data storage solutions in Snowflake and Data Lakes.
• Design and implement automated workflows using Azure Data Factory.

Skills

Python

PySpark

SQL

Data orchestration tools

Data Lakes

OpenShift

Containerized applications

Cloud platforms (Azure, AWS, GCP)

Data security and governance

Problem-solving skills

Education

Bachelor’s or Master’s degree in Computer Science or related field

Tools

Snowflake

Azure Data Factory

Kubernetes

Kafka

Hadoop

Job description

The Data Engineer will design, develop, and maintain scalable data pipelines and workflows using Python, PySpark, and cloud technologies.

This role involves managing data storage solutions, orchestrating ETL processes, and deploying containerized applications to support enterprise data initiatives.

Key Responsibilities

• Develop, construct, test, and maintain scalable data pipelines and ETL/ELT workflows using Python, PySpark, and SQL.
• Manage and optimize data storage solutions in Snowflake and Data Lakes.
• Integrate data from multiple sources while ensuring data quality, integrity, and security.
• Design and implement automated workflows using Azure Data Factory or similar orchestration tools.
• Deploy and manage containerized applications and services on OpenShift.
• Collaborate with Data Scientists, Analysts, and stakeholders to understand data requirements.
• Monitor and troubleshoot data pipelines, resolving performance bottlenecks.
• Maintain documentation of data architecture, workflows, and best practices.
• Ensure compliance with data governance and security policies.

Required Qualifications

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related field.
• Minimum 10 years of overall experience in data engineering or similar roles.
• Strong proficiency in Python and PySpark.
• Expertise in SQL and experience with Snowflake Data Warehouse.
• Hands-on experience with data orchestration tools such as Azure Data Factory.
• Knowledge of Data Lakes architecture and management.
• Experience deploying and managing containerized applications on OpenShift or Kubernetes.
• Familiarity with cloud platforms (Azure, AWS, GCP).
• Understanding of data security, governance, and compliance best practices.
• Strong problem-solving skills and attention to detail.
• Excellent communication and teamwork abilities.

Preferred Qualifications

• Experience with big data tools (e.g., Kafka, Hadoop).
• Knowledge of CI/CD pipelines and DevOps practices.
• Familiarity with additional cloud data services and tools.