Big Data Developer

Galent; Toronto; Hybrid; CAD 110,000 - 160,000; Full time

Job summary

Galent is seeking a Senior Backend Developer in Toronto, hybrid work 3/4 days onsite, to drive big data pipelines and API integrations. You will design, build, and maintain scalable systems in an enterprise environment.

You should have 5+ years in big data engineering, experience with Spark, Hadoop/CDP, Hive, and API-driven data exchanges. The role emphasizes AI-assisted development and efficient data processing at scale.

Qualifications

• 5+ years of experience in backend and big data engineering.
• Experience with API integration and AI-assisted development.
• Ability to design, build and maintain scalable data pipelines.

Responsibilities

• Design and develop Spark-Scala applications for large-scale data processing on Hadoop/CDP clusters
• Build and optimize ETL/ELT pipelines using Spark DataFrames, Datasets and Spark SQL
• Tune Spark jobs for performance (partitioning, caching, broadcast joins, shuffle optimization)
• Migrate Spark 2 applications to Spark 3 on Cloudera CDP platforms
• Work with Parquet, ORC, Avro file formats on HDFS
• Write complex HiveQL / Spark SQL queries including window functions, CTEs, subqueries and aggregations
• Design and maintain Hive external/managed tables and partitioned datasets
• Optimize slow-running queries and resolve correlated subquery issues
• Work with HDFS encryption zones and data governance requirements
• Develop and maintain bash shell scripts for job orchestration and automation
• Handle error management, return codes, logging and alerting in shell scripts
• Manage HDFS operations (hdfs dfs commands), file transfers, and data validation
• Build scripts and pipelines to extract data from REST APIs using curl and Python
• Parse and process JSON API responses and load into HDFS/Hive
• Manage pagination, error handling and retry logic for API calls
• Work with enterprise API gateways and URL parameter construction
• Leverage AI coding assistants to accelerate development
• Use AI tools for code review, SQL generation, script debugging and documentation
• Contribute to AI-assisted data quality and anomaly detection pipelines
• Explore and implement LLM-based automation for repetitive data engineering tasks
• Schedule and manage jobs using AAP / Control-M / cron
• Build and maintain Ansible playbooks for automated deployments
• Manage deployment pipelines including artifact versioning, Vault secret injection and environment-specific configuration
• Monitor job health, handle failures and implement alerting

Skills

SQL

Unix/Shell Scripting

Tools

Spark

Hadoop/CDP

Hive

HDFS

Job description

Location : Toronto, ON (Hybrid- 3/4 days onsite)

Top 3 skills required for this role:

• SQL
• Unix/Shell Scripting

Role Overview

We are looking for a Senior Backend Developer with 5+ years of experience in big data engineering, API integration, and AI-assisted development. The ideal candidate will design, build, and maintain scalable data pipelines and backend systems in a enterprise environment.

Key Responsibilities

• Design and develop Spark-Scala applications for large-scale data processing on Hadoop/CDP clusters
• Build and optimize ETL/ELT pipelines using Spark DataFrames, Datasets and Spark SQL
• Tune Spark jobs for performance (partitioning, caching, broadcast joins, shuffle optimization)
• Migrate Spark 2 applications to Spark 3 on Cloudera CDP platforms
• Work with Parquet, ORC, Avro file formats on HDFS
• Write complex HiveQL / Spark SQL queries including window functions, CTEs, subqueries and aggregations
• Design and maintain Hive external/managed tables and partitioned datasets
• Optimize slow-running queries and resolve correlated subquery issues
• Work with HDFS encryption zones and data governance requirements

Unix / Shell Scripting

• Develop and maintain bash shell scripts for job orchestration and automation
• Handle error management, return codes, logging and alerting in shell scripts
• Manage HDFS operations (hdfs dfs commands), file transfers, and data validation

API Extraction & Integration

• Build scripts and pipelines to extract data from REST APIs using curl and Python
• Parse and process JSON API responses and load into HDFS/Hive
• Manage pagination, error handling and retry logic for API calls
• Work with enterprise API gateways and URL parameter construction
• Leverage GitHub Copilot / AI coding assistants to accelerate development
• Use AI tools for code review, SQL generation, script debugging and documentation
• Contribute to AI-assisted data quality and anomaly detection pipelines
• Explore and implement LLM-based automation for repetitive data engineering tasks

Scheduling & Orchestration

• Schedule and manage jobs using AAP (Ansible Automation Platform) / Control-M / cron
• Build and maintain Ansible playbooks for automated deployments
• Manage deployment pipelines including artifact versioning, Vault secret injection and environment-specific configuration
• Monitor job health, handle failures and implement alerting

Nice to Have

• Experience with Cloudera CDP (7.x) and migration from HDP
• Knowledge of Kerberos, Vault, HDFS encryption zones
• Familiarity with CI/CD pipelines (Helios, GitHub Actions)
• Experience with MSSQL / JDBC connectivity from Spark
• Understanding of AML / Financial regulatory data domains.

We are an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex (including pregnancy, sexual orientation, or gender identity), national origin, citizenship status, age, disability, genetic information, protected veteran status, or any other characteristic protected by applicable law. https://www.e-verify.gov/sites/default/files/everify/posters/IER_RighttoWorkPoster.pdf