Data Engineer (Data Lake)

Comrise; Taguig; Remote; PHP 480,000 - 800,000; Full time

Job summary

Comrise is seeking a CRB Data Lake Support Associate to join the Data, Acquisition, Validation, and Enrichment team. You will help ingest, enrich, and distribute data across pipelines, collaborating with global teams and ensuring data flows remain uninterrupted.

You will work with Python, PySpark notebooks, and Azure Cloud tools to design, implement, and monitor data solutions, while developing your technical expertise within CRB’s data ecosystem.

Qualifications

• Proficiency in Python and PySpark notebooks.
• Experience designing and evolving data solutions in Azure Cloud.
• Experience with large data sets and data integration for analytics.
• Experience with Azure SQL DB or Synapse Analytics (or AWS equivalents).
• Experience presenting data using Power BI for quality reviews.
• Knowledge of Data Modeling and data retrieval for consumption.
• DevOps experience with Git and deployment to production.

Responsibilities

• Support data lake pipelines and monitoring of data flows.
• Configure deployments and monitoring for pipelines across regions.
• Troubleshoot pipeline failures and propose improvements.
• Review Synapse notebooks and communicate issues to development teams.
• Provide onboarding and configuration support for new data sources.
• Assist DevOps deployments to core data lake repositories.
• Apply security access governance in line with company practices.
• Produce and update Access reports quarterly.

Skills

Python

PySpark notebooks

Azure Cloud tools

Azure SQL DB

Azure Synapse Analytics

Power BI

Data Modeling

DevOps (Git)

Collaboration

Problem solving

Tools

Azure Cloud

Power BI

Git

Job description

Create your future.

As part of Corporate Risk and Broking (CRB) the Data, Acquisition, Validation, and Enrichment team focuses on ingesting data from various sources, enriching the data, storing the data, and serving the data to our various systems and stakeholders. We work heavily with other technical teams in CRB to ensure that what is collected is fit for use. The CRB Data Lake is a culmination of data sources with our CRB Data Ecosystem and plays a vital role in the collection, transformation, governance, and availability of data.

Explore an opportunity.

As a CRB Data Lake Support associate, you will work beside some of the companies’ top teams within CRB to facilitate the culmination of the strategic data in consumable packages and managing the distribution channels through pipelines and monitoring, while you progress toward becoming an experienced member of the team and CRB Technical community through your interactions with the data and teams.

You will have immediate exposure to real client data and technical teams which will draw on your imagination and creativity as well as your ability to analyze data, manage the data flows related to distribution to draw conclusions and present results of capacity, usage, availability. These experiences will help build your technical knowledge and overall industry expertise. You will be trained in how to plan, manage, prioritize, and communicate effective resolutions of the pipelines to ensure continued data flows.

Love your work.

You will partner with colleagues to provide support functions on the CRB Data Lake:

• Configure Deployment pipelines relate to code releases from various technical teams in CRB


• Setup monitoring schedules for pipelines deployed based on geographic areas of consumer usage


• Troubleshoot failed pipelines within the processes or guidelines, and suggesting solutions to resolve for future


• Review Synapse notebooks when errors occur and provide details on issues to development teams


• Review Synapse notebooks when errors occur and fix through development DevOps process


• Provide onboarding assistance and configuration for new ingestion data sources in conjunction with established processes and guidelines, and suggest improvements


• Provide DevOps support for deployments to core data lake repositories and monitoring


• Application of security access governance processes in line with company practices


• Review and updates to Access reports quarterly and ensure appropriate acceptance or removal is required


• Troubleshoot pipeline failures and report status on returned functionality


• Working with existing data models, data assets, within a data quality framework to suggest improvements for pipelines and data extraction/ingestion from sources


• Utilizing ways to visualize and distribute the data for business outcomes based on the pipeline and data processed functions

About you.

• Experience with Python and creating/edit pySpark notebooks


• Experience in designing, implementing and evolving data solutions within the Azure Cloud tools


• Experience in working with large data sets and co-mingling the data for business/system consumption as it relates to the Azure Cloud Modern Data Warehouse or similar data structures


• Experience with Azure SQL DB, Azure Synapse Analytics, or similar tools sets (e.g. AWS)


• Experience with data presentation to the user using reporting tools like Power BI for various data quality, presentations, and confidence reviews


• knowledge of Data Modeling and Management techniques, as it relates to retrieving data from systems to present for consumption.


• Dev-ops experience using GIT, developing, deploying code to production


• Able to manage multiple tasks with as required to review and resolve issues


• Able to collaborate with multiple teams to meet needs on a daily basis


• Able to work with a globally distributed team


• Ability to solve problems through critical thinking techniques and decision trees

This is a remote position; you will be working from your home