Data Engineer

Spait Infotech; Canada; On-site; CAD 90,000 - 130,000; Full time

Job summary

Spait Infotech is seeking a data engineer to design and maintain scalable data pipelines and ETL workflows. You will extract data from diverse sources, process it with Python, SQL, and PySpark, and build data warehouses and lakes. Collaboration with data science and BI teams is essential.

You will implement Airflow workflows, ensure data quality, and optimize pipelines across cloud platforms. Experience with Spark/Hadoop, databases, and CI/CD tooling is preferred.

Qualifications

• Strong knowledge of Python, SQL, and PySpark.

Responsibilities

• Design, develop, and maintain scalable data pipelines.
• Build and manage ETL/ELT workflows.
• Extract, transform, and load data from multiple sources.
• Develop data processing solutions using Python, SQL, and PySpark.
• Work with structured and unstructured data.
• Design and maintain data warehouses and data lakes.
• Develop automated workflows using Apache Airflow.
• Integrate data from APIs, databases, and external systems.
• Ensure data quality, accuracy, and consistency.
• Monitor, troubleshoot, and optimize data pipelines.
• Implement data validation, governance, and security.
• Collaborate with Data Scientists, Data Analysts, and BI teams.

Skills

Python

SQL

PySpark

ETL/ELT

Airflow

Kafka

Data warehousing

Git

Linux

Shell scripting

Tools

Apache Airflow

Apache Spark

Hadoop

Git

GitHub

Docker

Kubernetes

Linux

Shell

Snowflake

Databricks

Redshift

BigQuery

MySQL

PostgreSQL

MongoDB

AWS

Azure

GCP

AWS Glue

Azure Data Factory

Job description

• Design, develop, and maintain scalable data pipelines.
• Build and manage ETL/ELT workflows.
• Extract, transform, and load data from multiple sources.
• Develop data processing solutions using Python, SQL, and PySpark.
• Work with structured and unstructured data.
• Design and maintain data warehouses and data lakes.
• Develop automated workflows using Apache Airflow.
• Integrate data from APIs, databases, and external systems.
• Ensure data quality, accuracy, and consistency.
• Monitor, troubleshoot, and optimize data pipelines.
• Implement data validation, governance, and security.
• Collaborate with Data Scientists, Data Analysts, and BI teams.

Preferred Skills

• Strong knowledge of Python, SQL, and PySpark.
• Experience with Apache Spark and Hadoop.
• Knowledge of ETL/ELT processes.
• Hands-on experience with Apache Airflow.
• Knowledge of Apache Kafka and real-time data processing.
• Experience with MySQL, PostgreSQL, and MongoDB.
• Knowledge of Snowflake, Databricks, Redshift, or BigQuery.
• Experience with AWS, Azure, or GCP cloud platforms.
• Knowledge of AWS Glue and Azure Data Factory.
• Understanding of Data Warehousing and Data Modelling.
• Experience with Data Lakes and Lakehouse architecture.
• Knowledge of Git, GitHub, and CI/CD pipelines.
• Familiarity with Docker and Kubernetes.
• Knowledge of Linux and Shell scripting.
• Strong problem-solving and analytical skills.