Data Engineer (Azure Experience)

E-Outsource Asia; Kuala Lumpur; On-site; MYR 150,000 - 230,000; Full time

Job summary

E-Outsource Asia in Kuala Lumpur is seeking a Senior Data Engineer to design, build and optimize data pipelines on Azure Databricks. You will lead data integration across batch and streaming sources, create curated analytics datasets and ensure governance and quality across the platform.

You'll work with cross-functional teams, implement CI/CD for data jobs, and contribute to platform roadmaps while maintaining SLAs/SLOs. Fluent English and Mandarin helpful for client engagement.

Qualifications

• 3+ to 5+ years of delivery and design leadership.
• Hands-on Databricks experience (PySpark/SQL).
• Strong SQL and production Python experience.
• Experience with Azure data services (ADF, ADLS Gen2).
• Familiarity with Delta Lake and Lakehouse architectures.
• CI/CD and data pipeline automation experience.
• Fluent English and Mandarin for client communication.

Responsibilities

• Design, develop, test, deploy and maintain ETL/ELT pipelines on Azure.
• Ingest and integrate data from batch and streaming sources into ADLS Gen2.
• Build curated datasets for analytics and ML.
• Implement data quality checks, tests, and validation gates.
• Monitor pipelines, logging, alerting and RCA.
• Optimize pipeline performance and cost.
• Implement CI/CD for data jobs and infra.
• Ensure data governance, lineage and security; register assets.
• Collaborate with cross-functional teams; define SLAs/SLOs.
• Contribute to runbooks, docs, coding standards, and platform roadmap.

Skills

Databricks

PySpark

SQL

Python

Azure Data Factory

ADLS Gen2

Delta Lake

Lakehouse

CI/CD

Data governance

Data quality

Power BI

Looker

Tableau

Purview

English

Mandarin

Education

Bachelor’s degree in Information Systems / Computer Science / Data Management or related field

Job description

• Design, develop, test, deploy and maintain ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks.
• Implement scalable data processing and transformation logic in PySpark and SQL on Databricks (Delta Lake / Lakehouse patterns).
• Ingest and integrate data from diverse batch and streaming sources into ADLS Gen2 and Delta Lake.
• Build curated, documented datasets for analytics and ML (support dimensional models where appropriate).
• Implement automated data quality checks, unit/integration tests, and validation gates.
• Monitor pipeline health, implement logging, alerting and dashboards; perform RCA and production incident remediation.
• Optimise pipeline performance and cost cluster sizing, job scheduling, partitioning, caching and query tuning.
• Implement and maintain CI/CD for data jobs and infrastructure (Terraform/ARM/Bicep, Azure DevOps or GitHub Actions).
• Ensure data governance, lineage and security best practices; register assets in a data catalogue (e.g., Microsoft Purview or equivalent).
• Collaborate with cross-functional teams to collect requirements, define SLAs/SLOs and deliver production-ready solutions.
• Contribute to runbooks, documentation, coding standards and data platform roadmap.
• The following are the additional responsibilities for Senior Data Engineers:
• Lead design reviews, define platform standards and approve infra decisions.
• Drive cost-control initiatives and SLA reporting.
• Participate in vendor selection, proof-of-concepts and platform architecture decisions.

Profile

• Bachelor’s degree in Information Systems, Computer Science, Data Management,or related field (or equivalent experience).
• 3+ or 5+ years, with demonstrable leadership in delivery and design.
• Hands‑on Databricks experience (PySpark, SQL) and familiarity with Databricks runtimes.
• Strong experience with Azure data services: Azure Data Factory, ADLS Gen2.
• Strong SQL skills and production Python experience.
• Experience with CI/CD
• Experience applying data quality testing, monitoring and observability in production.
• Strong debugging, performance tuning, and troubleshooting skills.
• Good communication skills; experience working in cross‑functional teams.
• Fluent English and Mandarin.
• Experience with Delta Lake, Lakehouse architectures and ACID semantics.
• Familiarity with data cataloguing/governance tools (Microsoft Purview, Alation).
• Certifications: Databricks Certified Data Engineer, Microsoft DP-203 or Azure certifications.
• Experience with BI tools (Power BI, Looker, Tableau) and consuming datasets foranalytics.
• Fluent English and Mandarin to communicate with client teams based in China and Hong Kong.