Data Science Developer - Senior JP210

P@thlion Staffing Careers; Fredericton; On-site; CAD 70,000 - 110,000; Full time

Job summary

P@thlion Staffing Careers is seeking a skilled data engineering professional to join our cloud-centered team in Fredericton. The role focuses on architecting and implementing cloud-based data and analytics products that meet enterprise standards.

You will design and maintain data lakehouse structures, automate pipelines, and build analytics models while collaborating with IT and business partners to ensure effective use of cloud data solutions across projects.

Qualifications

• Experience with cloud-based data and analytics platforms (Azure preferred).
• Design and maintain cloud data lake/lakehouse structures and automated pipelines.
• Proficient in Python and SQL for analytics development.

Responsibilities

• Participate in product teams to analyze systems requirements, architect, design, code and implement cloud-based data and analytics products that conform to standards
• Design, create, and maintain cloud-based data lake and lakehouse structures, automated data pipelines, analytics models
• Liaises with cluster IT colleagues to implement products, conduct reviews, resolve operational problems, and support business partners in effective use of cloud-based data and analytics products.
• Analyze complex technical issues, identify alternatives and recommend solutions.
• Support the migration of legacy data pipelines from Azure Synapse Analytics and Azure Data Factory to modernized Databricks-based solutions leveraging Delta Lake and native orchestration capabilities
• Support the development of standards and a reusable framework that streamlines pipeline creation
• Participate in code reviews and prepare/conduct knowledge transfer to maintain code quality, promote team knowledge sharing, and enforce development standards across collaborative data projects.

Skills

Azure

Databricks

Python

SQL

CI/CD

GitHub

Agile

Data Lakehouse

Orchestration

Tools

Azure Data Factory

Databricks

Delta Lake

Unity Catalog

Job description

Responsibilities

• Participate in product teams to analyze systems requirements, architect, design, code and implement cloud-based data and analytics products that conform to standards
• Design, create, and maintain cloud-based data lake and lakehouse structures, automated data pipelines, analytics models
• Liaises with cluster IT colleagues to implement products, conduct reviews, resolve operational problems, and support business partners in effective use of cloud-based data and analytics products.
• Analyze complex technical issues, identify alternatives and recommend solutions.
• Support the migration of legacy data pipelines from Azure Synapse Analytics and Azure Data Factory (including stored procedures, views used by BI teams, and Parquet files in Azure Data Lake Storage (ADLS)) to modernized Databricks-based solutions leveraging Delta Lake and native orchestration capabilities
• Support the development of standards and a reusable framework that streamlines pipeline creation
• Participate in code reviews and prepare/conduct knowledge transfer to maintain code quality, promote team knowledge sharing, and enforce development standards across collaborative data projects.

General Skills

• Experience in multiple cloud-based data and analytics platforms and coding/programming/scripting tools to create, maintain, support and operate cloud-based data and analytics products, with a preference for Microsoft Azure
• Experience with designing, creating and maintaining cloud-based data lake and lakehouse structures, automated data pipelines, analytics models in real world implementations
• Strong background in building and orchestrating data pipelines using services like Azure Data Factory and Databricks
• Demonstrated ability to organize and manage data in a lakehouse following medallion architecture;
• Background with Databricks Unity Catalog for governance is a plus
• Proficient in using Python and SQL for data engineering and analytics development
• Familiar with CI/CD practices and tools for automating deployment of data solutions and managing code lifecycle
• Comfortable conducting and participating in peer code reviews in GitHub to ensure quality, consistency, and best practices
• Experience in assessing client information technology needs and objectives
• Experience in problem-solving to resolve complex, multi-component failures
• Experience in preparing knowledge transfer documentation and conducting knowledge transfer
• Experience working on an Agile team