Data Science Developer - Senior (ONSITE) JP210

Pathlion

Toronto

On-site

CAD 90,000 - 120,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Pathlion is seeking a skilled Data Engineer to join our cloud data platform team in Fredericton, NB. You will analyze requirements, design and implement lakehouse solutions and automated pipelines using Azure and Databricks.

You will collaborate with IT and business partners, migrate legacy pipelines to modern Delta Lake-based offerings, and contribute to standards and knowledge transfer in an Agile environment.

Qualifications

  • Experience with multiple cloud data platforms, with a preference for Microsoft Azure.
  • Experience designing and maintaining cloud-based data lake and lakehouse structures and automated pipelines.
  • Strong background building data pipelines using Azure Data Factory and Databricks.
  • Ability to organize data in a lakehouse following medallion architecture.
  • Databricks Unity Catalog governance knowledge is a plus.
  • Proficiency in Python and SQL for data engineering.
  • Familiarity with CI/CD practices for deploying data solutions.
  • Experience conducting peer code reviews in GitHub and promoting best practices.
  • Experience assessing client IT needs and troubleshooting complex issues.
  • Experience documenting knowledge transfer and working in Agile teams.

Responsibilities

  • Participate in product teams to analyze systems requirements, architect, design, code and implement cloud-based data and analytics products that conform to standards
  • Design, create, and maintain cloud-based data lake and lakehouse structures, automated data pipelines, analytics models
  • Liaises with cluster IT colleagues to implement products, conduct reviews, resolve operational problems, and support business partners in effective use of cloud-based data and analytics products.
  • Analyze complex technical issues, identify alternatives and recommend solutions.
  • Support the migration of legacy data pipelines from Azure Synapse Analytics and Azure Data Factory to modernized Databricks-based solutions leveraging Delta Lake and native orchestration capabilities
  • Support the development of standards and a reusable framework that streamlines pipeline creation
  • Participate in code reviews and prepare/conduct knowledge transfer to maintain code quality, promote team knowledge sharing, and enforce development standards across collaborative data projects.

Skills

Azure
Databricks
Python
SQL
CI/CD
GitHub
Agile
Data Lake
Medallion Architecture
Unity Catalog

Tools

Azure Data Factory
CI/CD Tools
GitHub

Job description

Responsibilities
  • Participate in product teams to analyze systems requirements, architect, design, code and implement cloud-based data and analytics products that conform to standards
  • Design, create, and maintain cloud-based data lake and lakehouse structures, automated data pipelines, analytics models
  • Liaises with cluster IT colleagues to implement products, conduct reviews, resolve operational problems, and support business partners in effective use of cloud-based data and analytics products.
  • Analyze complex technical issues, identify alternatives and recommend solutions.
  • Support the migration of legacy data pipelines from Azure Synapse Analytics and Azure Data Factory (including stored procedures, views used by BI teams, and Parquet files in Azure Data Lake Storage (ADLS)) to modernized Databricks-based solutions leveraging Delta Lake and native orchestration capabilities
  • Support the development of standards and a reusable framework that streamlines pipeline creation
  • Participate in code reviews and prepare/conduct knowledge transfer to maintain code quality, promote team knowledge sharing, and enforce development standards across collaborative data projects.
General Skills
  • Experience in multiple cloud-based data and analytics platforms and coding/programming/scripting tools to create, maintain, support and operate cloud-based data and analytics products, with a preference for Microsoft Azure
  • Experience with designing, creating and maintaining cloud-based data lake and lakehouse structures, automated data pipelines, analytics models in real world implementations
  • Strong background in building and orchestrating data pipelines using services like Azure Data Factory and Databricks
  • Demonstrated ability to organize and manage data in a lakehouse following medallion architecture;
  • Background with Databricks Unity Catalog for governance is a plus
  • Proficient in using Python and SQL for data engineering and analytics development
  • Familiar with CI/CD practices and tools for automating deployment of data solutions and managing code lifecycle
  • Comfortable conducting and participating in peer code reviews in GitHub to ensure quality, consistency, and best practices
  • Experience in assessing client information technology needs and objectives
  • Experience in problem-solving to resolve complex, multi-component failures
  • Experience in preparing knowledge transfer documentation and conducting knowledge transfer
  • Experience working on an Agile team
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Science Developer - Senior JP210
Data Science Developer - Senior JP210

P@thlion Staffing Careers • Fredericton

On-site
CAD 70,000 - 110,000
Senior Data Science Developers
Senior Data Science Developers

HRBrain Inc. • Toronto

Hybrid
CAD 90,000 - 140,000
Hybrid work model
Senior Databricks Data Engineer
Senior Databricks Data Engineer

Raise - find a more meaningful working experience • Toronto

Hybrid
CAD 100,000 - 167,000
Sr. Data Engineer
Sr. Data Engineer

Recruiting in Motion • Toronto

Hybrid
CAD 120,000 - 140,000
Principal Data Engineer – Databricks
Principal Data Engineer – Databricks

Astra-North Infoteck Inc. • Toronto

On-site
CAD 140,000 - 190,000
Databricks Engineer
Databricks Engineer

Compunnel Inc. • Mississauga

On-site
CAD 120,000 - 150,000
Lead/Senior Data Engineer
Lead/Senior Data Engineer

Celebal Technologies • Ottawa

On-site
CAD 110,000 - 160,000
Databricks Architect
Databricks Architect

TechDoQuest • Toronto

On-site
CAD 100,000 - 130,000
Azure Data Engineer
Azure Data Engineer

Astra-North Infoteck Inc. • Toronto

Hybrid
CAD 100,000 - 130,000
Senior Databricks Engineer
Senior Databricks Engineer

KData Inc. • Toronto

On-site
CAD 170,000 - 210,000