Sr Data Engineer

MPA Recruitment; Halifax; On-site; CAD 80,000 - 110,000; Full time

Job summary

A leading recruitment firm based in Canada is seeking a Senior Data Engineer. In this role, you will design and develop large-scale data processing software within agile teams, ensuring operational readiness and code quality. You should have strong development skills in Java, Scala, or Python, and experience with Big Data technologies. This position offers the opportunity to lead and mentor while contributing to impactful data solutions.

Qualifications

• Experience in developing large-scale data processing solutions.
• Ability to write quality, testable code and experience with automated testing.
• Awareness of distributed data processing frameworks and serialization formats.

Responsibilities

• Develop data processing software for Big Data technologies.
• Collaborate with Architects to ensure software meets requirements.
• Coach and mentor team members.

Skills

Experience with FHIR

Strong software development experience in Java, Scala, or Python

Proficient in SQL for analytical queries

Solid understanding of ETL / ELT data processing pipelines

Experience with Continuous Integration and Continuous Deployment

Tools

AWS

Azure

GCP

Databricks

Kafka

Job description

Overview

As a Senior Data Engineer (Senior Associate), you will be responsible for designing and developing data processing and data persistence software components for solutions that handle data at scale. Working in agile teams, Senior Data Engineers provide strong development leadership and take responsibility for significant technical components of data systems. You will work within a multi-skilled agile team to design and develop large-scale data processing software to meet user needs in demanding production environments.

Responsibilities

• Develop data processing software primarily for deployment in Big Data technologies. The role encompasses the full software lifecycle including design, code, test and defect resolution.
• Collaborate with Architects and Lead Engineers to ensure the software supports non-functional requirements.
• Collaborate with colleagues to resolve implementation challenges and ensure code quality and maintainability; lead by example in code quality.
• Work with operations teams to ensure operational readiness.
• Advise customers and managers on estimated effort and technical implications of user stories and journeys.
• Coach and mentor team members.

Minimum (Essential) Requirements

• Experience with FHIR
• Strong software development experience in one of Java, Scala, or Python
• Software development experience with data-processing platforms from vendors such as AWS, Azure, GCP, Databricks
• Experience of developing substantial components for large-scale data processing solutions and deploying into a production environment
• Proficient in SQL and SQL extensions for analytical queries
• Solid understanding of ETL / ELT data processing pipelines and design patterns
• Aware of key features and pitfalls of distributed data processing frameworks, data stores and data serialization formats
• Able to write quality, testable code and has experience of automated testing
• Experience with Continuous Integration and Continuous Deployment techniques

Desirable Requirements

• Experience of performance tuning
• Experience of data visualisation and complex data transformations
• Experience with streaming and event-processing architectures including technologies such as Kafka and change-data-capture (CDC) products
• Expertise in continuous improvement and sharing input on data best practice
• Practical experience with AI technologies, tools, processes and delivery

Seniority level

• Mid-Senior level

Employment type

• Full-time

Job function

• Engineering, Consulting, and Information Technology

Industries

• IT Services and IT Consulting, IT System Custom Software Development, and IT System Design Services