Data Engineer

Manulife Insurance Malaysia; Toronto; On-site; CAD 80,000 - 100,000; Full time

Job summary

A leading company is seeking a skilled Data Engineer to enhance their data capabilities. The role involves developing data pipelines, ensuring data quality, and collaborating with teams to drive strategic business decisions. Ideal candidates will have a strong background in data engineering and ETL processes, with proficiency in Azure tools and programming languages.

Qualifications

• 5+ years of experience as a Data Engineer.
• Proficient in ETL/ELT logic and data mart design.

Responsibilities

• Design and manage end-to-end data pipelines.
• Collaborate with teams for data mapping and integration.
• Implement data quality assurance and governance.

Skills

Data Engineering

ETL Processes

Data Integration

Problem Solving

Communication

Education

Bachelor’s Degree in Computer Science

Master's Degree

Tools

Azure Data Factory

Databricks

Python

Java

Scala

Power BI

SQL

Job description

job requisition id JR25011611

The Opportunity

Manulife is embarking on replacing and building new data capabilities to help fuel our bold ambition to become a digital customer leader. We are seeking a skilled and motivated data engineer to join our dynamic team and play a key role in implementing, optimizing, and maintaining assets that deliver these capabilities. The ideal candidate possesses a solid background in data engineering, ETL processes, and data integration, with a passion for understanding data to drive strategic business decisions.

Responsibilities

1. Data Pipeline Development: Design, develop, and manage end-to-end data pipelines that facilitate the detailed extraction, transformation, and loading of data from diverse sources.
2. Data Mapping & Integration: Collaborate closely with multi-functional teams to understand and design schemas for data from various source systems and other transactional or application databases, ensuring accuracy and reliability.
3. ETL Optimization: Continuously improve and optimize ETL processes to enhance data flow efficiency, minimize latency, and support real-time and batch processing requirements.
4. Data Transformation: Implement data cleansing, enrichment, and transformation processes to ensure high-quality data is available for analysis and reporting.
5. Data Quality Assurance: Design testing plans, develop and implement data quality checks, validation rules, and supervising mechanisms to maintain data accuracy and integrity.
6. Platform Enhancement: Collaborate with various technical resources from across the organization to identify and implement enhancements to the infrastructure, integrations, and functionalities.
7. Data Architecture: Work closely with business leads and data architects to design, implement, and manage end-to-end architecture based on business requirements.
8. User Documentation: Create and maintain comprehensive documentation for data pipelines, processes, and configurations to facilitate knowledge sharing and onboarding.
9. Collaboration: Partner with other data engineers, data analysts, business collaborators, and data scientists to understand data requirements and translate them into effective data engineering solutions.
10. Performance Monitoring: Monitor data pipeline performance and solve issues to ensure optimal data flow and proactively find opportunities for enhancement.
11. Data Governance and Compliance: Ensure consistency with data privacy and compliance standards throughout the data lifecycle.

What motivates you?

• You obsess about customers, listen, engage and act for their benefit.
• You think big, with curiosity to discover ways to use your agile approach and enable business outcomes.
• You thrive in teams and enjoy getting things done together.
• You take ownership and build solutions, focusing on what matters.
• You do what is right, work with integrity and speak up.
• You share your humanity, helping us build a diverse and inclusive work environment for everyone.

What we are looking for

• Bachelor’s Degree in Computer Science, Information Technology, or a related field. Master's degree is a plus.
• 5+ years of experience as a Data Engineer, with a track record of efficiently implementing and maintaining data pipelines, scheduling, monitoring, notification, and ETL processes using Azure Data Factory, Databricks, Python/Pyspark, Java, Scala.
• Understanding of Azure infrastructure; subscriptions, resource groups, resources, access control with RBAC (role-based access control), integrations with Azure AD and Azure security principles (user group, service principal, managed identity), network concepts (VNet, Subnet, NSG rules, private endpoints), password/credential/key management and data protection.
• Experience deploying and integrating Azure Data Services (Azure Data Factory, Databricks) using DevOps tools and principles: GitHub Repository, Jenkins CI/CD pipelines, integrated unit tests, etc.
• Knowledge of Azure Data Lake Storage (ADLS Gen2) and its topology on blob storage with file hierarchy, storage account, containers, and folders.
• Proficient in data mart fact and dimension design concepts, ETL/ELT logic to perform upsert and type-2, and applying/implementing with Python/Pyspark/Stored Procedures/SQL in ADLS (with Lakehouse architecture using Databricks) or in Azure Synapse (with Dedicated SQL pool).
• Experience using Power BI to connect to sources with ADLS (including Delta Lake), Databricks SQL, Azure Synapse (SQL pool), design semantic layers/data models, create/publish reporting content (datasets/paginated reports/interactive dashboards), and manage workspaces.
• Solid understanding of data privacy and compliance regulations and standard methodologies.
• Excellent problem-solving skills and the ability to solve technical issues efficiently.
• Effective communication skills to collaborate with technical and non-technical partners.
• Thorough approach with a commitment to delivering high-quality work in a fast-paced environment.