Senior Data Engineer - Python, PySpark and OpenShift Lead

Compunnel, Inc.; Mississauga; On-site; CAD 100,000 - 130,000; Full time

Job summary

A leading data engineering firm is seeking a Data Engineer to design and maintain scalable data pipelines using Python and PySpark. This role requires managing data storage solutions, orchestrating ETL processes, and deploying containerized applications. The ideal candidate will have extensive experience in SQL, data orchestration tools, and cloud platforms. Strong communication and teamwork abilities are essential, along with a solid understanding of data security and governance.

Qualifications

• Minimum 10 years of overall experience in data engineering or similar roles.
• Expertise in SQL and experience with Snowflake Data Warehouse.
• Strong proficiency in Python and PySpark.

Responsibilities

• Develop, construct, test, and maintain scalable data pipelines.
• Manage and optimize data storage solutions in Snowflake and Data Lakes.
• Design and implement automated workflows using Azure Data Factory.

Skills

Python

PySpark

SQL

Data orchestration tools

Data Lakes

OpenShift

Containerized applications

Cloud platforms (Azure, AWS, GCP)

Data security and governance

Problem-solving skills

Education

Bachelor’s or Master’s degree in Computer Science or related field

Tools

Snowflake

Azure Data Factory

Kubernetes

Kafka

Hadoop

Job description

A leading data engineering firm is seeking a Data Engineer to design and maintain scalable data pipelines using Python and PySpark. This role requires managing data storage solutions, orchestrating ETL processes, and deploying containerized applications. The ideal candidate will have extensive experience in SQL, data orchestration tools, and cloud platforms. Strong communication and teamwork abilities are essential, along with a solid understanding of data security and governance.