Data Engineer

E-Outsource Asia; Kuala Lumpur; Hybrid; MYR 180,000 - 300,000; Full time

Job summary

E- Outsource Asia is seeking a Senior Data Engineer to design, implement, and optimize end-to-end data pipelines on Azure Databricks in Kuala Lumpur. You will lead data processing, model ingestion, and governance for analytics and ML initiatives, collaborating with cross-functional teams in a fast-paced environment.

The role requires strong PySpark/SQL, Azure Data Factory expertise, and a track record of CI/CD for data platforms. Mandarin and English proficiency is a plus for client interactions.

Qualifications

• Bachelor’s degree in Information Systems, Computer Science, Data Management, or related field is required.
• 3+ or 5+ years of leadership in delivery and design, as stated in the profile.

Responsibilities

• Design, develop, test, deploy and maintain ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks.
• Implement scalable data processing and transformation logic in PySpark and SQL on Databricks.
• Ingest and integrate data from diverse batch and streaming sources into ADLS Gen2 and Delta Lake.
• Build curated datasets for analytics and ML with documented datasets.
• Implement automated data quality checks, unit/integration tests, and validation gates.
• Monitor pipeline health, logging, alerting, RCA and production incident remediation.
• Optimize pipeline performance, cost, scheduling, partitioning, caching and query tuning.
• Implement and maintain CI/CD for data jobs and infrastructure.
• Ensure data governance, lineage and security; register assets in data catalogue.
• Lead design reviews, define platform standards, drive cost-control initiatives, SLA reporting, vendor selections and architecture decisions.

Skills

Databricks PySpark SQL

Python

CI/CD

Data quality monitoring

Performance tuning

Communication

English & Mandarin

Delta Lake / Lakehouse

Data cataloguing governance

BI tools (Power BI Looker Tableau)

Databricks certifications

Education

Bachelor’s degree in Information Systems or equivalent

Tools

Terraform

ARM/Bicep

Azure DevOps

GitHub Actions

Job description

• Design, develop, test, deploy and maintain ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks.
• Implement scalable data processing and transformation logic in PySpark and SQL on Databricks (Delta Lake / Lakehouse patterns).
• Ingest and integrate data from diverse batch and streaming sources into ADLS Gen2 and Delta Lake.
• Build curated, documented datasets for analytics and ML (support dimensional models where appropriate).
• Implement automated data quality checks, unit/integration tests, and validation gates.
• Monitor pipeline health, implement logging, alerting and dashboards; perform RCA and production incident remediation.
• Optimise pipeline performance and cost cluster sizing, job scheduling, partitioning, caching and query tuning.
• Implement and maintain CI/CD for data jobs and infrastructure (Terraform/ARM/Bicep, Azure DevOps or GitHub Actions).
• Ensure data governance, lineage and security best practices; register assets in a data catalogue (e.g., Microsoft Purview or equivalent).
• Collaborate with cross-functional teams to collect requirements, define SLAs/SLOs and deliver production-ready solutions.
• Contribute to runbooks, documentation, coding standards and data platform roadmap.
• The following are the additional responsibilities for Senior Data Engineers:
• Lead design reviews, define platform standards and approve infra decisions.
• Drive cost-control initiatives and SLA reporting.
• Participate in vendor selection, proof-of-concepts and platform architecture decisions.

Profile

• Bachelor’s degree in Information Systems, Computer Science, Data Management,or related field (or equivalent experience).
• 3+ or 5+ years, with demonstrable leadership in delivery and design.
• Hands-on Databricks experience (PySpark, SQL) and familiarity with Databricks runtimes.
• Strong experience with Azure data services: Azure Data Factory, ADLS Gen2.
• Strong SQL skills and production Python experience.
• Experience with CI/CD
• Experience applying data quality testing, monitoring and observability in production.
• Strong debugging, performance tuning, and troubleshooting skills.
• Good communication skills; experience working in cross-functional teams.
• Fluent English and Mandarin.
• Experience with Delta Lake, Lakehouse architectures and ACID semantics.
• Familiarity with data cataloguing/governance tools (Microsoft Purview, Alation).
• Certifications: Databricks Certified Data Engineer, Microsoft DP-203 or Azure certifications.
• Experience with BI tools (Power BI, Looker, Tableau) and consuming datasets foranalytics.
• Fluent English and Mandarin to communicate with client teams based in China and Hong Kong.