Senior Data Engineer (Spark, Databricks, Python, MDM)

CMC-APAC Private Limited; Santo Niño 1st; On-site; PHP 1,200,000 - 1,800,000; Full time

Job summary

CMC-APAC Private Limited is seeking a Senior Data Engineer to enhance data ingestion and processing for the Asset Data Warehouse. You will build scalable pipelines and APIs, collaborating with data scientists and analysts to drive data quality and actionable insights.

The role requires 5+ years in data engineering, experience with data lakes/warehouses, MDM, Spark, Kafka, and cloud platforms (Azure/GCP/AWS). Proficiency with Delta Lake and Databricks is desirable.

Qualifications

• Minimum 5 years in a data engineer role.
• Experience with data lake, data warehouse and data lake house architectures.
• Familiar with MDM processes and related tools.

Responsibilities

• Create and manage master records for business entities with data accuracy.
• Implement data governance, data quality, profiling and lineage.
• Build robust data pipelines across Cloud, Private Data Centre and Hybrid ecosystems.
• Assemble large, complex data sets from diverse sources.
• Collaborate with data scientists, ML engineers and business users to derive insights.
• Automate workflows and redesign infrastructure for scalability.
• Support cross-functional teams in a dynamic environment.

Skills

Data engineering

Data governance

MDM

Delta Lake

Databricks

Spark

Kafka

Cloud platforms

Data quality

Data lakes

Tools

Hadoop

SQL

Databricks

Delta Lake

Job description

About the role

Our Client is looking for a Senior Data Engineer to improve data ingestion and data processing jobs and create new microservice APIs as part of ADWH (Asset Data Warehouse) enhancements.

Key responsibilities

• Create and manage a single master record for each business entity, ensuring data consistency, accuracy, and reliability

• Implement data governance processes, including data quality management, data profiling, data remediation, and automated data lineage

• Create and maintain multiple robust and high-performance data processing pipelines within Cloud, Private Data Centre, and Hybrid data ecosystems

• Assemble large, complex data sets from a wide variety of data sources

• Collaborate with Data Scientists, Machine Learning Engineers, Business Analysts, and Business users to derive actionable insights on data quality

• Design and implement internal processes to automate manual workflows, optimize data delivery, and re-design infrastructure for greater scalability

• Support and work with cross-functional teams in a dynamic environment

About you

• At least 5 years of experience in a data engineer role

• Familiar with data lake, data warehouse and data lake house architectures

• Familiar with MDM processes such as golden record creation, survivorship, reconciliation, enrichment, and quality

• Experience building and operating large-scale data lakes and data warehouses

• Experience with Hadoop ecosystem and big data tools, including Spark and Kafka

• Experience with Master Data Management (MDM) tools and platforms such as Informatica MDM, Talend Data Catalog, Semarchy xDM, IBM PIM & IKC, or Profisee

• Experience in data governance, including data quality management, data profiling, data remediation, and automated data lineage

• Experience with stream-processing systems including Spark-Streaming

• Experience working with Cloud services using one or more Cloud providers such as Azure, GCP, or AWS

• Experience with Delta Lake and Databricks