Cloud Data & Analytics Engineer (Databricks/Azure)

Pathlion; Toronto; On-site; CAD 90,000 - 120,000; Full time

Job summary

Pathlion is seeking a skilled Data Engineer to join our cloud data platform team in Fredericton, NB. You will analyze requirements, design and implement lakehouse solutions and automated pipelines using Azure and Databricks.

You will collaborate with IT and business partners, migrate legacy pipelines to modern Delta Lake-based offerings, and contribute to standards and knowledge transfer in an Agile environment.

Qualifications

• Experience with multiple cloud data platforms, with a preference for Microsoft Azure.
• Experience designing and maintaining cloud-based data lake and lakehouse structures and automated pipelines.
• Strong background building data pipelines using Azure Data Factory and Databricks.
• Ability to organize data in a lakehouse following medallion architecture.
• Databricks Unity Catalog governance knowledge is a plus.
• Proficiency in Python and SQL for data engineering.
• Familiarity with CI/CD practices for deploying data solutions.
• Experience conducting peer code reviews in GitHub and promoting best practices.
• Experience assessing client IT needs and troubleshooting complex issues.
• Experience documenting knowledge transfer and working in Agile teams.

Responsibilities

• Participate in product teams to analyze systems requirements, architect, design, code and implement cloud-based data and analytics products that conform to standards
• Design, create, and maintain cloud-based data lake and lakehouse structures, automated data pipelines, analytics models
• Liaises with cluster IT colleagues to implement products, conduct reviews, resolve operational problems, and support business partners in effective use of cloud-based data and analytics products.
• Analyze complex technical issues, identify alternatives and recommend solutions.
• Support the migration of legacy data pipelines from Azure Synapse Analytics and Azure Data Factory to modernized Databricks-based solutions leveraging Delta Lake and native orchestration capabilities
• Support the development of standards and a reusable framework that streamlines pipeline creation
• Participate in code reviews and prepare/conduct knowledge transfer to maintain code quality, promote team knowledge sharing, and enforce development standards across collaborative data projects.

Skills

Azure

Databricks

Python

SQL

CI/CD

GitHub

Agile

Data Lake

Medallion Architecture

Unity Catalog

Tools

Azure Data Factory

CI/CD Tools

GitHub

Job description

Pathlion is seeking a skilled Data Engineer to join our cloud data platform team in Fredericton, NB. You will analyze requirements, design and implement lakehouse solutions and automated pipelines using Azure and Databricks.

You will collaborate with IT and business partners, migrate legacy pipelines to modern Delta Lake-based offerings, and contribute to standards and knowledge transfer in an Agile environment.