Remote Data Lake Engineer: Build & Optimize Pipelines

Comrise; Metro Manila; Remote; PHP 450,000 - 750,000; Full time

Job summary

Comrise is hiring a CRB Data Lake Support associate to collaborate with top teams, build and monitor data pipelines, and package strategic data for distribution. You will work with real client data and gain exposure to Azure-based data solutions while contributing to data quality and governance.

You will partner with colleagues to support deployments, troubleshooting, and ingestion of sources, while applying DevOps practices and cross-team collaboration in a remote, global setting.

Qualifications

• Experience with Python and creating/edit pySpark notebooks.
• Experience in designing, implementing and evolving data solutions within the Azure Cloud tools.
• Experience with Azure SQL DB, Azure Synapse Analytics, or similar tools sets (e.g. AWS).
• Experience with data presentation to the user using reporting tools like Power BI.
• Knowledge of Data Modeling and Management techniques for retrieving data from systems.
• Dev-ops experience using GIT, developing, deploying code to production.

Responsibilities

• Configure Deployment pipelines relate to code releases from various technical teams in CRB.
• Setup monitoring schedules for pipelines deployed based on geographic areas of consumer usage.
• Troubleshoot failed pipelines within the processes or guidelines, and suggesting solutions to resolve for future.
• Review Synapse notebooks when errors occur and provide details on issues to development teams.
• Review Synapse notebooks when errors occur and fix through development DevOps process.
• Provide onboarding assistance and configuration for new ingestion data sources in conjunction with established processes and guidelines, and suggest improvements.
• Provide DevOps support for deployments to core data lake repositories and monitoring.
• Application of security access governance processes in line with WTW practices.
• Review and updates to Access reports quarterly and ensure appropriate acceptance or removal is required.
• Troubleshoot pipeline failures and report status on returned functionality.
• Working with existing data models, data assets, within a data quality framework to suggest improvements for pipelines and data extraction/ingestion from sources.
• Utilizing ways to visualize and distribute the data for business outcomes based on the pipeline and data processed functions.

Skills

Python

PySpark

Critical thinking

Collaboration

Remote work

Tools

Azure Cloud

Azure Synapse

Azure SQL DB

Power BI

Git

DevOps

Automation

Job description

Comrise is hiring a CRB Data Lake Support associate to collaborate with top teams, build and monitor data pipelines, and package strategic data for distribution. You will work with real client data and gain exposure to Azure-based data solutions while contributing to data quality and governance.

You will partner with colleagues to support deployments, troubleshooting, and ingestion of sources, while applying DevOps practices and cross-team collaboration in a remote, global setting.