Senior Data Platform Engineer - Azure Databricks

E-Outsource Asia; Kuala Lumpur; Hybrid; MYR 180,000 - 300,000; Full time

Job summary

E- Outsource Asia is seeking a Senior Data Engineer to design, implement, and optimize end-to-end data pipelines on Azure Databricks in Kuala Lumpur. You will lead data processing, model ingestion, and governance for analytics and ML initiatives, collaborating with cross-functional teams in a fast-paced environment.

The role requires strong PySpark/SQL, Azure Data Factory expertise, and a track record of CI/CD for data platforms. Mandarin and English proficiency is a plus for client interactions.

Qualifications

• Bachelor’s degree in Information Systems, Computer Science, Data Management, or related field is required.
• 3+ or 5+ years of leadership in delivery and design, as stated in the profile.

Responsibilities

• Design, develop, test, deploy and maintain ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks.
• Implement scalable data processing and transformation logic in PySpark and SQL on Databricks.
• Ingest and integrate data from diverse batch and streaming sources into ADLS Gen2 and Delta Lake.
• Build curated datasets for analytics and ML with documented datasets.
• Implement automated data quality checks, unit/integration tests, and validation gates.
• Monitor pipeline health, logging, alerting, RCA and production incident remediation.
• Optimize pipeline performance, cost, scheduling, partitioning, caching and query tuning.
• Implement and maintain CI/CD for data jobs and infrastructure.
• Ensure data governance, lineage and security; register assets in data catalogue.
• Lead design reviews, define platform standards, drive cost-control initiatives, SLA reporting, vendor selections and architecture decisions.

Skills

Databricks PySpark SQL

Python

CI/CD

Data quality monitoring

Performance tuning

Communication

English & Mandarin

Delta Lake / Lakehouse

Data cataloguing governance

BI tools (Power BI Looker Tableau)

Databricks certifications

Education

Bachelor’s degree in Information Systems or equivalent

Tools

Terraform

ARM/Bicep

Azure DevOps

GitHub Actions

Job description

E- Outsource Asia is seeking a Senior Data Engineer to design, implement, and optimize end-to-end data pipelines on Azure Databricks in Kuala Lumpur. You will lead data processing, model ingestion, and governance for analytics and ML initiatives, collaborating with cross-functional teams in a fast-paced environment.

The role requires strong PySpark/SQL, Azure Data Factory expertise, and a track record of CI/CD for data platforms. Mandarin and English proficiency is a plus for client interactions.