Sr Data Engineer

MPA Recruitment; Halifax; On-site; CAD 120,000 - 180,000; Full time

Job summary

MPA Recruitment is seeking a Senior Data Engineer in Halifax, Canada, to design and build data processing and persistence components for scale. You will lead in agile teams, shaping the software lifecycle from design to production, and collaborate with architects and operations to ensure reliable, high-quality data systems.

You will work with Java/Scala/Python, SQL, and big data platforms on AWS/Azure/GCP/Databricks, focusing on robust ETL/ELT pipelines and distributed data processing.

Qualifications

• Strong software development experience in Java, Scala, or Python.
• Experience with data-processing platforms on AWS, Azure, GCP, or Databricks.
• Designing large-scale data processing solutions deployed to production.
• Proficiency in SQL and analytical queries.
• Understanding ETL/ELT pipelines and data processing patterns.
• Knowledge of distributed data processing frameworks and data stores.
• Ability to write quality, testable code with automated testing.
• Experience with CI/CD techniques.

Responsibilities

• Lead development of data processing software for scale and production readiness.
• Collaborate with Architects and Lead Engineers to support non-functional requirements.
• Contribute to code quality and maintainability; lead by example.
• Coach and mentor team members in data engineering practices.
• Advise stakeholders on effort estimates and technical implications.

Skills

Java/Scala/Python

SQL

ETL/ELT pipelines

CI/CD

Cloud big data platforms

Distributed data processing

Tools

Docker

Kubernetes

Apache NiFi

Kafka

Job description

As a Senior Data Engineer, you will be responsible or designing and developing data processing and data persistence software components for solutions which handle data at scale. Working in agile teams, Senior Data Engineers provide strong development leadership and take responsibility for significant technical components of data systems. You will work within a multi-skilled agile team to design and develop large-scale data processing software to meet user needs in demanding production environments.

Your responsibilities will include working to develop data processing software primarily for deployment in Big Data technologies. The role encompasses the full software lifecycle including design, code, test and defect resolution. Working with Architects and Lead Engineers to ensure the software supports non-functional needs. Collaborating with colleagues to resolve implementation challenges and ensure code quality and maintainability remains high. Leads by example in code quality. Working with operations teams to ensure operational readiness. Advising customers and managers on the estimated effort and technical implications of user stories and user journeys. Coaching and mentoring team members.

It is a fast-paced environment, so it is important for you to make sound, reasoned decisions. You will do this whilst learning about new technologies and approaches, with talented colleagues that will help you to develop and grow. You will support your colleagues and more junior colleagues, providing direction support as you solve challenging problems together.

MINIMUM (ESSENTIAL) REQUIREMENTS

• Strong software development experience in one of Java, Scala, or Python
• Software development experience with data-processing platforms from vendors such as AWS, Azure, GCP, Databricks.
• Experience of developing substantial components for large-scale data processing solutions and deploying into a production environment
• Proficient in SQL and SQL extensions for analytical queries
• Solid understanding of ETL/ELT data processing pipelines and design patterns
• Aware of key features and pitfalls of distributed data processing frameworks, data stores and data serialisation formats
• Able to write quality, testable code and has experience of automated testing
• Experience with Continuous Integration and Continuous Deployment techniques

DESIRABLE:

• Experience of data pipeline development using Apache Nifi
• Experience of Docker and Kubernetes
• Experience of performance tuning
• Experience of data visualisation and complex data transformations
• Experience with steaming and event-processing architectures including technologies such as Kafka and change-data-capture (CDC) products
• Expertise in continuous improvement and sharing input on data best practice