Senior Big Data Engineer: Spark, Hadoop & API Pipelines

Galent; Toronto; On-site; CAD 120,000 - 180,000; Full time

Job summary

Galent in Toronto is seeking a Senior Backend Developer to design, build, and maintain scalable data pipelines and backend systems in an enterprise environment. You will lead Spark-Scala projects on Hadoop/CDP clusters, optimize ETL pipelines, and migrate Spark 2 apps to Spark 3 while ensuring data governance and security.

The role requires 5+ years of experience in big data engineering, API integration, and AI-assisted development, with hands-on work on Spark, Hive, HDFS, and CI/CD tooling.

Qualifications

• 5+ years of backend or data engineering experience in enterprise environments.
• Hands-on with Spark, Hadoop, and data lake ecosystems.
• Experience migrating Spark 2 to Spark 3 and building scalable ETL pipelines.

Responsibilities

• Design and develop Spark-Scala apps for large-scale data processing on Hadoop/CDP clusters.
• Build and optimize ETL/ELT pipelines using Spark DataFrames, Datasets and Spark SQL.
• Tune Spark jobs for performance (partitioning, caching, broadcast joins, shuffle optimization).
• Migrate Spark 2 apps to Spark 3 on Cloudera CDP platforms.
• Work with Parquet, ORC, Avro on HDFS.
• Write complex HiveQL / Spark SQL queries with window functions, CTEs, subqueries and aggregations.
• Design and maintain Hive external/managed tables and partitioned datasets.
• Optimize slow-running queries and resolve correlated subquery issues.
• Work with HDFS encryption zones and data governance requirements.
• Develop and maintain bash scripts for job orchestration and automation.
• Handle error management, logging and alerting in shell scripts.
• Manage HDFS operations (hdfs dfs commands), file transfers, and data validation.
• Build scripts and pipelines to extract data from REST APIs using curl and Python.
• Parse and process JSON API responses and load into HDFS/Hive.
• Manage pagination, error handling and retry logic for API calls.
• Work with enterprise API gateways and URL parameter construction.
• Leverage GitHub Copilot / AI coding assistants to accelerate development.
• Use AI tools for code review, SQL generation, script debugging and documentation.
• Contribute to AI-assisted data quality and anomaly detection pipelines.
• Explore and implement LLM-based automation for repetitive data engineering tasks.

Skills

Big data engineering

API integration

AI-assisted development

Tools

Spark

Hadoop

Hive

HDFS

CDP

Spark SQL

Parquet/ORC/Avro

Job description

Galent in Toronto is seeking a Senior Backend Developer to design, build, and maintain scalable data pipelines and backend systems in an enterprise environment. You will lead Spark-Scala projects on Hadoop/CDP clusters, optimize ETL pipelines, and migrate Spark 2 apps to Spark 3 while ensuring data governance and security.

The role requires 5+ years of experience in big data engineering, API integration, and AI-assisted development, with hands-on work on Spark, Hive, HDFS, and CI/CD tooling.