Principal Data Engineer (Azure)

Tiger Analytics

Toronto

Hybrid

CAD 90,000 - 120,000

Full time

14 days+

Get more replies from employers

Send a job-specific resume in minutes.

Benefits offered by this job

Significant career development opportunities

Flexible work environment

Competitive compensation packages

Job summary

A leading analytics consulting firm in Toronto is seeking a Principal Data Engineer to work on Azure Data technologies. You will design scalable data pipelines, collaborate across teams, and tackle various big data challenges. The ideal candidate should have strong skills in Azure Data Factory, PySpark, and Databricks. This role offers significant career growth opportunities in a dynamic environment.

Qualifications

Experience in implementing Data Lake with Azure technologies.
Strong debugging skills in SQL, Python, or Scala/Java.
Experience with big data technologies like Hadoop and Spark.

Responsibilities

Design and build scalable data ingestion pipelines for batch and streaming datasets.
Collaborate with teams to build analytical solutions.
Handle data harmonization and validate pipelines.

Skills

Azure Data Factory (ADF)

PySpark

Databricks

Azure SQL Database

Hadoop

Spark

Kafka

SQL

Python

Education

Data Engineering certification on Microsoft Azure (DP-203)

Databricks Certified Developer (DE)

Tools

Azure DevOps

Git

Jenkins

Job description

1 month ago Be among the first 25 applicants

Get AI-powered advice on this job and more exclusive features.

Tiger Analytics is a global AI and analytics consulting firm. With data and technology at the core of our solutions, we are solving problems that eventually impact the lives of millions globally. Our culture is modeled around expertise and respect with a team-first mindset. Headquartered in Silicon Valley, you'll find our delivery centers across the globe and offices in multiple cities across India, the US, UK, Canada, and Singapore, including a

substantial remote global workforce.

We're Great Place to Work-Certified. Working at Tiger Analytics, you'll be at the heart of an AI revolution. You'll work with teams that push the boundaries of what is possible and build solutions that energize and inspire.

Requirements

Curious about the role? What your typical day would look like?

As a Principal Data Engineer (Azure), you would have hands on experience working on Azure as cloud, Databricks and some exposure/experience on Data Modelling. You will build and learn about a variety of analytics solutions & platforms, data lakes, modern data platforms, data fabric solutions, etc. using different Open Source, Big Data, and Cloud technologies on Microsoft Azure.

Design and build scalable & metadata-driven data ingestion pipelines (For Batch and Streaming Datasets)
Conceptualize and execute high-performance data processing for structured and unstructured data, and data

harmonization

Schedule, orchestrate, and validate pipelines
Design exception handling and log monitoring for debugging
Ideate with your peers to make tech stack and tools-related decisions
Interact and collaborate with multiple teams (Consulting/Data Science & App Dev) and various stakeholders to meet deadlines, to bring Analytical Solutions to life

What do we expect?

Experience in implementing Data Lake with technologies like Azure Data Factory (ADF), PySpark, Databricks, ADLS,

Azure SQL Database

A comprehensive foundation with working knowledge of Azure Synapse Analytics, Event Hub & Streaming

Analytics, Cosmos DB, and Purview

A passion for writing high-quality code and the code should be modular, scalable, and free of bugs (debugging

skills in SQL, Python, or Scala/Java).

Enthuse to collaborate with various stakeholders across the organization and take complete ownership of

deliverables.

Experience in using big data technologies like Hadoop, Spark, Airflow, NiFi, Kafka, Hive, Neo4J, Elastic Search
Adept understanding of different file formats like Delta Lake, Avro, Parquet, JSON, and CSV
Good knowledge of building and designing REST APIs with real-time experience working on Data Lake or

Lakehouse projects.

Experience in supporting BI and Data Science teams in consuming the data in a secure and governed manner
Certifications like Data Engineering on Microsoft Azure (DP-203) or Databricks Certified Developer (DE) are

valuable addition.

Note: The designation will be commensurate with expertise and experience. Compensation packages are among the best in the industry.

Job Requirement

Mandatory: Azure Data Factory (ADF), PySpark, Databricks, ADLS, Azure SQL Database
Optional: Azure Synapse Analytics, Event Hub & Streaming Analytics, Cosmos DB and Purview
Strong programming, unit testing & debugging skills in SQL, Python or Scala/Java
Some experience of using big data technologies like Hadoop, Spark, Airflow, NiFi, Kafka, Hive, Neo4J, Elastic Search
Good Understanding of different file formats like Delta Lake, Avro, Parquet, JSON and CSV
Experience of working in Agile projects and following DevOps processes with technologies like Git, Jenkins & Azure DevOps
Good to have:
Experience of working on Data Lake & Lakehouse projects
Experience of building REST services and implementing service-oriented architectures
Experience of supporting BI and Data Science teams in consuming the data in a secure and governed manner
Certifications like Data Engineering on Microsoft Azure (DP-203) or Databricks Certified Developer (DE)

Benefits

This position offers an excellent opportunity for significant career development in a fast-growing and challenging entrepreneurial environment with a high degree of individual responsibility.

Seniority level