Sr Data Engineer

Mpa Recruitment; Halifax; On-site; CAD 90,000 - 130,000; Full time

Job summary

Mpa Recruitment is seeking a Senior Data Engineer (Senior Associate) to design and develop data processing and persistence software for big data applications in a production environment. You will work in an agile team, provide technical leadership on major data components, and collaborate with architects to meet non-functional requirements.

Responsibilities include building data processing software for deployment on platforms like AWS, Azure, GCP, or Databricks, guiding implementation with

Qualifications

• Experience with FHIR.
• Strong software development in Java, Scala, or Python.
• Experience with data-processing platforms from cloud vendors (AWS, Azure, GCP) or Databricks.
• Experience delivering substantial components for large-scale data processing in production.
• Proficient in SQL and analytical SQL extensions.
• Understanding of ETL/ELT pipelines and design patterns.
• Familiar with distributed data processing frameworks and data stores.
• Ability to write quality, testable code and automated tests.
• Experience with CI/CD.

Responsibilities

• Design and develop data processing and persistence software for big data solutions.
• Lead software components and provide technical leadership in agile teams.
• Ensure software meets non-functional requirements with architects and lead engineers.
• Collaborate to resolve implementation challenges and maintain code quality.
• Work with operations to ensure production readiness.
• Advise stakeholders on estimation and technical implications of stories.
• Mentor teammates and promote best practices.

Skills

FHIR

Java

Scala

Python

Big Data platforms (AWS/Azure/GCP/Data

Tools

Databricks

SQL

Kafka

Job description

As a Senior Data Engineer (Senior Associate) , you will be responsible or designing and developing data processing and data persistence software components for solutions which handle data at scale. Working in agile teams, Senior Data Engineers provide strong development leadership and take responsibility for significant technical components of data systems. You will work within a multi-skilled agile team to design and develop large-scale data processing software to meet user needs in demanding production environments.

Your responsibilities will include:

• Working to develop data processing software primarily for deployment in Big Data technologies. The role encompasses the full software lifecycle including design, code, test and defect resolution.
• Working with Architects and Lead Engineers to ensure the software supports non-functional needs.
• Collaborating with colleagues to resolve implementation challenges and ensure code quality and maintainability remains high. Leads by example in code quality.
• Working with operations teams to ensure operational readiness
• Advising customers and managers on the estimated effort and technical implications of user stories and user journeys.
• Coaching and mentoring team members.

MINIMUM (ESSENTIAL) REQUIREMENTS:

• Experience with FHIR
• Strong software development experience in one of Java, Scala, or Python
• Software development experience withdata-processing platforms from vendors such as AWS, Azure, GCP, Databricks.
• Experience of developing substantial components for large-scale data processing solutions and deploying into a production environment
• Proficient in SQL and SQL extensions for analytical queries
• Solid understanding of ETL/ELT data processing pipelines and design patterns
• Aware of key features and pitfalls of distributed data processing frameworks, data stores and data serialisation formats
• Able to write quality, testable code and has experience of automated testing
• Experience with Continuous Integration and Continuous Deployment techniques
• A keen interest in AI Technologies

DESIRABLE REQUIREMENTS:

• Experience of performance tuning
• Experience of data visualisation and complex data transformations
• Experience with steaming and event-processing architectures including technologies such as Kafka and change-data-capture (CDC) products
• Expertise in continuous improvement and sharing input on data best practice
• Practical experience with AI technologies, tools, processes and delivery.