Data Ingestion Engineer

Brainhunter Systems Ltd

Toronto

On-site

CAD 90,000 - 130,000

Full time

4 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Canada Life Data team seeks a Data Ingestion Engineer to design, implement, and maintain pipelines that ingest data from diverse sources into our data platforms using IDMC on Azure, aided by Data Factory and Databricks.

You will work with cross-functional teams to ensure data quality, monitoring, and scalable operations while supporting incident response, governance, and security best practices.

Qualifications

  • 5+ years in data operations or development (ETL, Data Warehouse).
  • 2+ years of recent development experience in IDMC, MS-Azure, and Databricks.
  • Experience designing and implementing data ingestion systems with MS-Azure/Databricks.
  • Experience with Kafka, CI/CD and CDC tools preferred.

Responsibilities

  • Design, develop, test, and support end-to-end ingestion pipelines using IDMC in MS Azure data platform that supports structured and unstructured data from multiple sources using Azure Data Factory and Databricks.
  • Collaborate with cross-functional teams to understand data requirements and design efficient ingestion patterns and workflows.
  • Implement and maintain controls and alerts to monitor data pipelines and proactively identify data issues; ensure data quality and integrity.
  • Collaborate with support teams to triage incidents and resolve major incidents, outages, or disruptions.
  • Automate processes to improve reliability, efficiency, and scalability using MS Azure tools.
  • Conduct performance and capacity assessments and develop plans to improve performance and reduce costs.
  • Focus on data governance, compliance controls, and security measures to protect sensitive data.
  • Apply best practices for scalable, reusable, and reliable data pipelines, and document configurations and procedures.

Skills

ETL/Data Warehouse
IDMC
Azure/Data Factory
Databricks
Kafka
CI/CD
CDC
Python
SQL
Monitoring pipelines

Tools

Informatica IDMC
Azure Data Factory
Databricks
Kafka
CI/CD tooling

Job description

Anticipated Start Date: 4 weeks from the official offer day

Role Title: Data Ingestion Engineer

Reliability Clearance required: Yes, please submit only candidates who either hold a valid Reliability Clearance or are eligible to obtain one. Candidates who do not meet the required five‑year residency requirement should not be submitted.

Any specific tools/skillset:

  • 5+ years of prior experience in data operations or development role (e-g ETL, Data Warehouse) with 2+ years of recent development experience in IDMC, MS-Azure and Data bricks
  • Demonstrated experience in designing and implementing data ingestion systems using MS-Azure /Databricks
  • Experience in Kafka, CI/CD and CDC tools preferred
  • Solid programming experience with Python or SQL
  • Experience in setting up monitoring and controls for MS Azure data pipelines (prior experience with Data warehousing/ other data technologies is accepted)
  • Strong knowledge in service management including incident and change management procedures.
Skills and Attributes
  • Problem-solving skills and the ability to trouble shoot complex issues.
  • Contributing to positive team culture and promoting common goals, team success and business outcomes
  • Valuing and appreciating different views and bring in collaborative approach to achieving consensus.
  • Encourages inclusivity and a sense of unity within all operational areas of Canada Data team.

As a Data Ingestion Engineer, you will join a dynamic and high-performing Canada Life Data team dedicated to providing Data and AI solutions by ensuring the robustness, scalability, and reliability of data ingestion systems within our organization. Your key focus will be on designing, implementing, and maintaining the pipelines necessary for efficient and seamless data ingestion from various sources into our data platforms. You will partner with cross functional teams, enterprise, and portfolio architects to enhance operational performance, trouble shoot issues, and implement best practices for data reliability.

Key Responsibilities
  • Design, develop, test, and support end-to-end ingestion pipelines using Informatica IDMC in MS Azure data platform that supports the consumption of structured and unstructured data from various sources with the use of Azure Data Factory and Data Bricks
  • Collaborate with cross functional teams to understand data requirements, design efficient data ingestion patterns and workflows.
  • Implement and maintain controls and alerts to monitor data pipelines to proactively identify data related issues. Ensure data quality and integrity is maintained across the pipelines.
  • Collaborate with support teams to triage incidents to quickly resolve major incidents, system outages or disruptions.
  • Automate processes to support smooth operation of data pipelines to improve reliability, efficiency, scalability using the MS Azure tools.
  • Conduct performance and capacity assessments, identify bottlenecks, and develop plans to improve performance, and reduce operational cost.
  • Focused on implementing data governance and compliance controls, and security measure to protect sensitive data.
  • Apply best practices for creation and maintenance of data pipelines including scalability, reusability, and reliability.
  • Document system configurations, processes and troubleshooting procedures.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

ETL & Data Ingestion Engineer
ETL & Data Ingestion Engineer

Brainhunter Systems Ltd • Toronto

Hybrid
CAD 70,000 - 117,000
Hybrid work arrangement
Senior Azure Data Ingestion Engineer (IDMC/Databricks)
Senior Azure Data Ingestion Engineer (IDMC/Databricks)

Brainhunter Systems Ltd • Toronto

On-site
CAD 90,000 - 130,000
“Data Ingestion Engineer\"
“Data Ingestion Engineer\"

Brainhunter Systems Ltd • Toronto

Hybrid
CAD 90,000 - 130,000
Data Migration/ Modernization Lead
Data Migration/ Modernization Lead

Accenture PLC • Toronto

Hybrid
CAD 109,000 - 220,000
Senior Data Engineer
Senior Data Engineer

Cpus Engineering Staffing Solutions Inc. • Pickering

Hybrid
CAD 90,000 - 120,000
Data Engineer
Data Engineer

Hirevantis LLC • Toronto

Hybrid
CAD 105,000 - 135,000
Health benefits
Hybrid/Remote across Canada
Data Engineer (Databricks)
Data Engineer (Databricks)

Cpus Engineering Staffing Solutions Inc. • Oshawa

Hybrid
CAD 124,000 - 131,000
Senior Data Engineer
Senior Data Engineer

Michael Page CA • Winnipeg

On-site
CAD 90,000 - 120,000
Senior Data Platform Engineer
Senior Data Platform Engineer

Canada Life • London

Hybrid
CAD 85,000 - 135,000
Career development
Health & wellness benefits
Volunteer day
+3
OPEN: Data Engineer
OPEN: Data Engineer

Cpus Engineering Staffing Solutions Inc. • Oshawa

On-site
CAD 80,000 - 100,000