Intelligence Data Engineer (Azure Experience)

E-Outsource Asia; Kuala Lumpur; On-site; MYR 150,000 - 210,000; Full time

Job summary

E-Outsource Asia in Kuala Lumpur is seeking a Senior Data Engineer to design, develop, and maintain scalable ETL/ELT pipelines on Azure Databricks and ADF. You will build Lakehouse patterns, Delta Lake datasets, and production-grade data quality checks.

You will collaborate with cross-functional teams, optimize performance and costs, implement CI/CD for data jobs, and contribute to data governance with Purview or Alation.

Qualifications

• 3+ years of leadership in delivery and design.

Responsibilities

• Design, develop, test, deploy and maintain ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks.

Skills

Databricks PySpark

SQL

Python

Azure Data Factory

ADLS Gen2

CI/CD

Data quality checks

Performance tuning

Communication skills

Education

Bachelor’s degree in Information Systems/CS/Data Management

Tools

Power BI

Looker

Tableau

Microsoft Purview

Alation

Databricks runtimes

Job description

• Design, develop, test, deploy and maintain ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks.
• Implement scalable data processing and transformation logic in PySpark and SQL on Databricks (Delta Lake / Lakehouse patterns).
• Ingest and integrate data from diverse batch and streaming sources into ADLS Gen2 and Delta Lake.
• Build curated, documented datasets for analytics and ML (support dimensional models where appropriate).
• Implement automated data quality checks, unit/integration tests, and validation gates.
• Monitor pipeline health, implement logging, alerting and dashboards; perform RCA and production incident remediation.
• Optimise pipeline performance and cost cluster sizing, job scheduling, partitioning, caching and query tuning.
• Implement and maintain CI/CD for data jobs and infrastructure (Terraform/ARM/Bicep, Azure DevOps or GitHub Actions).
• Ensure data governance, lineage and security best practices; register assets in a data catalogue (e.g., Microsoft Purview or equivalent).
• Collaborate with cross-functional teams to collect requirements, define SLAs/SLOs and deliver production-ready solutions.
• Contribute to runbooks, documentation, coding standards and data platform roadmap.
• The following are the additional responsibilities for Senior Data Engineers:
• Lead design reviews, define platform standards and approve infra decisions.
• Drive cost-control initiatives and SLA reporting.
• Participate in vendor selection, proof-of-concepts and platform architecture decisions.

Profile

• Bachelor’s degree in Information Systems, Computer Science, Data Management,or related field (or equivalent experience).
• 3+ or 5+ years, with demonstrable leadership in delivery and design.
• Hands‑on Databricks experience (PySpark, SQL) and familiarity with Databricks runtimes.
• Strong experience with Azure data services: Azure Data Factory, ADLS Gen2.
• Strong SQL skills and production Python experience.
• Experience with CI/CD
• Experience applying data quality testing, monitoring and observability in production.
• Strong debugging, performance tuning, and troubleshooting skills.
• Good communication skills; experience working in cross‑functional teams.
• Fluent English and Mandarin.
• Experience with Delta Lake, Lakehouse architectures and ACID semantics.
• Familiarity with data cataloguing/governance tools (Microsoft Purview, Alation).
• Certifications: Databricks Certified Data Engineer, Microsoft DP-203 or Azure certifications.
• Experience with BI tools (Power BI, Looker, Tableau) and consuming datasets foranalytics.
• Fluent English and Mandarin to communicate with client teams based in China and Hong Kong.