Sr EDL Hadoop Data Engg or Sr Developer

Ness Digital Engineering; Canada; On-site; CAD 80,000 - 120,000; Full time

Job summary

A leading software engineering firm based in Canada is seeking experienced professionals in data engineering with strong expertise in the Hadoop ecosystem. Candidates should have proficiency in SQL, PySpark, and experience with cloud-native data lakes. Ideal for individuals with a robust understanding of data ingestion and transformation processes in large-scale environments. This position emphasizes collaboration and efficiency in data warehousing and ETL processes.

Qualifications

• Strong hands-on expertise in Hadoop ecosystem and related technologies.
• Proficiency in SQL, PySpark, Scala, or Java.
• Experience with distributed computing and data warehousing.
• Familiarity with cloud platforms and data governance tools.

Skills

Hadoop ecosystem (HDFS, Hive, Spark, Oozie, Yarn, HBase, Kafka, Zookeeper)

SQL

PySpark

Scala

Java

Cloud-native data lakes (AWS, Azure, GCP)

Data governance tools (Apache Atlas, Ranger, Collibra)

Workflow orchestration tools (Airflow, Oozie)

Data Warehousing and ETL processes

Jenkins

ServiceNow

Confluence

Bitbucket

JIRA

Job description

Required Skills & Experience

Technical Expertise

Technical Expertise

Strong hands-on expertise in Hadoop ecosystem (HDFS, Hive, Spark, Oozie, Yarn, HBase, Kafka, Zookeeper).

Deep understanding of data ingestion, transformation, and storage patterns in large-scale environments.

Experience with distributed computing, data partitioning, and parallel processing.

Proficiency in SQL , PySpark , Scala , or Java .

Familiarity with cloud-native data lakes on AWS (EMR, Glue, S3) , Azure (HDInsight, ADLS, Synapse) , or GCP (Dataproc, BigQuery).

Knowledge of data governance tools (Apache Atlas, Ranger, Collibra) and workflow orchestration tools (Airflow, Oozie).

• Expertise in Data Warehousing and ETL processes, including Design, Development, Support, Implementation, and Testing.
• Hands on exp in Architecture, design including requirement analysis, performance tuning, data conversion, loading, extraction, transformation, and creating job pipelines.
• Strong knowledge of the Retail Domain and experience with various stages of data warehouse projects, including data extraction, cleansing, aggregation, validation, transformation, and loading.
• Exp in using DataStage components such as Sequential File, Join, Sort, Merge, Lookup, Transformer, Remove Duplicates, Copy, Filter, Funnel, Dataset, Change Data Capture, and Aggregator.
• Strong at database commands (DDL and DML) and data warehousing implementations models.
• Hands on exp with the Hadoop ecosystem , including HDFS, Hive, Sqoop, NiFi, and YARN.
• Familiar with Mainframe ESP for job scheduling.
• Implementation exp in indexes, table partitioning, collections, analytical functions, and materialized views . Created and managed tables, views, constraints, and indexes.
• Experienced in CI / CD processes using Jenkins and SourceTree.
• Proficient with ServiceNow , Confluence, Bitbucket, and JIRA.

Preferred Qualifications

Experience integrating EDL with modern lakehouse platforms (Databricks, Snowflake, Synapse, BigQuery).

Understanding of machine learning pipelines and real-time analytics use cases.

Exposure to data mesh or domain-driven data architectures .

Certifications in Hadoop, Cloudera, AWS, or Azure data services.