Big Data Engineer (PySpark / Spark / Kafka)

Astra-North Infoteck Inc.

Toronto

Hybrid

CAD 100,000 - 150,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Astra-North Infoteck Inc. in Toronto seeks an experienced Data Engineer with deep Big Data expertise to design, develop, and support enterprise-scale data platforms.

The role focuses on PySpark, Apache Spark, Kafka, Hadoop ecosystem, and NiFi, covering ingestion, transformation, and real-time processing. You will build scalable pipelines, implement ETL/ELT workflows, ensure data quality and governance, and collaborate with cross-functional teams.

Qualifications

  • Hands-on experience with PySpark, Spark SQL, and DataFrames for large-scale data processing.
  • Strong knowledge of Hadoop components (HDFS, Hive, YARN) and data warehousing concepts.
  • Proficient SQL and Python for data ingestion, transformation, and validation.
  • Experience with real-time processing and streaming pipelines is a plus.

Responsibilities

  • Design, develop, and optimize scalable data pipelines using PySpark, Spark, Hadoop, and Apache NiFi.
  • Build batch and real-time data processing solutions and ensure data quality.
  • Develop and support Kafka-based streaming applications and event-driven architectures.
  • Create and optimize ETL/ELT workflows for large-scale structured and unstructured datasets.
  • Develop complex SQL queries for data extraction, transformation, validation, and troubleshooting.
  • Monitor, troubleshoot, and optimize Spark jobs and data pipelines.

Skills

PySpark
Apache Spark (Spark SQL, DataFrames)
Hadoop Ecosystem (HDFS, Hive, YARN)
SQL
Python

Tools

Jenkins
Bitbucket
Git
JIRA
Confluence

Job description

Work Mode: Hybrid – 3 Days a Week in Office

Job Overview

We are seeking an experienced Data Engineer with strong expertise in Big Data technologies to design, develop, and support enterprise-scale data platforms. The ideal candidate should have hands-on experience in PySpark, Apache Spark, Kafka, Hadoop ecosystem components, and Apache NiFi, along with a strong understanding of data ingestion, transformation, and real-time processing frameworks.

Key Responsibilities
  • Design, develop, and optimize scalable data pipelines using PySpark, Spark, Hadoop, and Apache NiFi.
  • Build and maintain batch and real-time data processing solutions.
  • Develop and support Kafka-based streaming applications and event-driven architectures.
  • Create and optimize ETL/ELT workflows for large-scale structured and unstructured datasets.
  • Develop complex SQL queries for data extraction, transformation, validation, and troubleshooting.
  • Implement data ingestion solutions from databases, APIs, files, and streaming sources.
  • Monitor, troubleshoot, and enhance the performance of Spark jobs and data pipelines.
  • Collaborate with architects, business analysts, and development teams to deliver high-quality data solutions.
  • Support platform upgrades, deployments, testing, certification, and production releases.
  • Ensure data quality, governance, security, and operational excellence across data platforms.
Mandatory Skills
  • PySpark
  • Apache Spark (Spark SQL, DataFrames)
  • Hadoop Ecosystem (HDFS, Hive, YARN)
  • SQL
  • Python
Preferred Skills
  • Spark Streaming
  • Hive
  • Scala
  • Jenkins, Bitbucket, Git
  • JIRA, Confluence
  • Data Warehousing Concepts and Dimensional Modeling
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Hybrid Big Data Engineer — PySpark, Spark & Kafka
Hybrid Big Data Engineer — PySpark, Spark & Kafka

Astra-North Infoteck Inc. • Toronto

Hybrid
CAD 100,000 - 150,000
Spark Developer (With Java)
Spark Developer (With Java)

Capgemini • Mississauga

On-site
CAD 90,000 - 130,000
Data Engineer
Data Engineer

Lancesoft • Montreal

On-site
CAD 90,000 - 120,000
Big Data Engineer
Big Data Engineer

KTek Resourcing • Mississauga

On-site
CAD 70,000 - 110,000
Senior Data Engineer
Senior Data Engineer

Bank of America Corporation • New Brunswick

On-site
CAD 90,000 - 130,000
Senior Python PySpark Data Engineer – Databricks
Senior Python PySpark Data Engineer – Databricks

Compunnel Inc. • Mississauga

Remote
CAD 90,000 - 120,000
Python/PySpark Developer
Python/PySpark Developer

Compunnel Inc. • Mississauga

Remote
CAD 90,000 - 120,000
Senior Data Engineer - Python, PySpark and OpenShift Lead
Senior Data Engineer - Python, PySpark and OpenShift Lead

Compunnel Inc. • Mississauga

On-site
CAD 100,000 - 130,000
Senior Software Engineer – Core Java, Apache Spark, Python, SQL, Data Processing
Senior Software Engineer – Core Java, Apache Spark, Python, SQL, Data Processing

Astra-North Infoteck Inc. • Montreal

On-site
CAD 90,000 - 130,000
Data Engineer Python/PySpark/OpenShift
Data Engineer Python/PySpark/OpenShift

Compunnel Inc. • Mississauga

On-site
CAD 100,000 - 130,000