Hybrid Big Data Engineer — PySpark, Spark & Kafka

Astra-North Infoteck Inc.

Toronto

Hybrid

CAD 100,000 - 150,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Astra-North Infoteck Inc. in Toronto seeks an experienced Data Engineer with deep Big Data expertise to design, develop, and support enterprise-scale data platforms.

The role focuses on PySpark, Apache Spark, Kafka, Hadoop ecosystem, and NiFi, covering ingestion, transformation, and real-time processing. You will build scalable pipelines, implement ETL/ELT workflows, ensure data quality and governance, and collaborate with cross-functional teams.

Qualifications

  • Hands-on experience with PySpark, Spark SQL, and DataFrames for large-scale data processing.
  • Strong knowledge of Hadoop components (HDFS, Hive, YARN) and data warehousing concepts.
  • Proficient SQL and Python for data ingestion, transformation, and validation.
  • Experience with real-time processing and streaming pipelines is a plus.

Responsibilities

  • Design, develop, and optimize scalable data pipelines using PySpark, Spark, Hadoop, and Apache NiFi.
  • Build batch and real-time data processing solutions and ensure data quality.
  • Develop and support Kafka-based streaming applications and event-driven architectures.
  • Create and optimize ETL/ELT workflows for large-scale structured and unstructured datasets.
  • Develop complex SQL queries for data extraction, transformation, validation, and troubleshooting.
  • Monitor, troubleshoot, and optimize Spark jobs and data pipelines.

Skills

PySpark
Apache Spark (Spark SQL, DataFrames)
Hadoop Ecosystem (HDFS, Hive, YARN)
SQL
Python

Tools

Jenkins
Bitbucket
Git
JIRA
Confluence

Job description

Astra-North Infoteck Inc. in Toronto seeks an experienced Data Engineer with deep Big Data expertise to design, develop, and support enterprise-scale data platforms.

The role focuses on PySpark, Apache Spark, Kafka, Hadoop ecosystem, and NiFi, covering ingestion, transformation, and real-time processing. You will build scalable pipelines, implement ETL/ELT workflows, ensure data quality and governance, and collaborate with cross-functional teams.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Big Data Engineer (PySpark / Spark / Kafka)
Big Data Engineer (PySpark / Spark / Kafka)

Astra-North Infoteck Inc. • Toronto

Hybrid
CAD 100,000 - 150,000
Senior Azure Data Engineer - Pipelines, Databricks & ML Ops
Senior Azure Data Engineer - Pipelines, Databricks & ML Ops

Astra-North Infoteck Inc. • Toronto

Hybrid
CAD 120,000 - 170,000
Spark Big Data Engineer | Scala/Java & ETL Analytics
Spark Big Data Engineer | Scala/Java & ETL Analytics

Tata Consultancy Services Limited • Toronto

On-site
CAD 90,000 - 110,000
Senior Data Engineer - Real-Time Data Platforms Lead
Senior Data Engineer - Real-Time Data Platforms Lead

Bank of America Corporation • New Brunswick

On-site
CAD 90,000 - 130,000
Senior Data Engineer - Big Data & Spark Expert
Senior Data Engineer - Big Data & Spark Expert

Wipro Ltd. • Mississauga

On-site
CAD 100,000 - 150,000
Senior AWS Data Engineer: Lake Formation & Data Platform
Senior AWS Data Engineer: Lake Formation & Data Platform

Astra-North Infoteck Inc. • Toronto

On-site
CAD 120,000 - 150,000
Data Engineering Software Engineer — Pipelines & Spark
Data Engineering Software Engineer — Pipelines & Spark

Unchain Data • Toronto

On-site
CAD 136,000 - 160,000
Bonus opportunities
Equity
Benefits
Senior Data Engineer, Databricks & Azure Expert
Senior Data Engineer, Databricks & Azure Expert

NTT America, Inc. • Toronto

On-site
CAD 89,000 - 111,000
Health and dental coverage
Remote work allowance
Retirement savings
+4
Senior Software Engineer: Big Data Spark & Java (Canada)
Senior Software Engineer: Big Data Spark & Java (Canada)

Capgemini Engineering • Canada

On-site
CAD 65,200 - 158,550
Paid time off
Medical, dental, and vision coverage
Retirement savings plans
+2
Senior Forward-Deployed AI Engineer: Real-Data Solutions
Senior Forward-Deployed AI Engineer: Real-Data Solutions

Astra-North Infoteck Inc. • Toronto

On-site
CAD 120,000 - 180,000