Data Engineering Tech Lead — Databricks & AWS

BP PLC; Kuala Lumpur; Hybrid; MYR 180,000 - 300,000; Full time

Benefits offered by this job

Life and health insurance

Learning opportunities

Job summary

BP in Kuala Lumpur is seeking a Senior Data Engineer to own the data domain within an autonomous squad. You will design, build, migrate, operate, and continuously improve scalable data products and pipelines using Databricks, PySpark, Spark SQL, and AWS, ensuring reliability, security and cost efficiency.

You will lead data ingestion, transformation, and storage architectures, enforce data quality and governance, and drive observability and incident resolution.

Qualifications

• 8+ years of experience in Data Engineering, Software Engineering, or related technical disciplines.
• 5+ years hands-on Python experience.
• 3+ years working with PySpark and distributed data processing.
• Strong hands-on experience with Databricks, PySpark, Spark SQL, SQL, Delta Lake and data modeling.
• Experience designing and operating enterprise-scale data ingestion, transformation, migration, and processing frameworks.
• Hands-on experience with Databricks Auto Loader, Delta Live Tables (DLT), Workflows, and Databricks Asset Bundles (DAB).
• Strong AWS experience, including IAM, S3 and integration with AWS cloud services.
• Experience with Unity Catalog including governance, metadata, lineage, and access control.
• Experience with CI/CD, Infrastructure as Code, automated testing, Git, and deployment automation.
• Strong understanding of observability, data quality, performance optimization, cost monitoring, reliability, and production support.

Responsibilities

• Lead the design, development, migration, and operation of scalable data pipelines and data products using Databricks, PySpark, Spark SQL, and AWS.
• Own the squad’s data engineering domain, providing technical direction and ensuring secure, reliable solutions.
• Design and implement scalable data ingestion and migration frameworks using Databricks Auto Loader and Delta Live Tables (DLT).
• Develop and maintain data models, transformation pipelines, and storage solutions with emphasis on data quality, integrity, performance, and scalability.
• Build and maintain pipeline orchestration and deployment capabilities using Databricks Workflows, Asset Bundles (DAB), CI/CD, and Infrastructure as Code.
• Integrate data platforms with the AWS ecosystem, including IAM, storage, security, networking, and related services.
• Implement governance, security, lineage, and access controls using Unity Catalog and enterprise standards.
• Establish observability, monitoring, alerting, and cost optimization across pipelines and Databricks workloads.
• Drive operational excellence through incident resolution, root cause analysis, automation, and reliability improvements.
• Provide technical leadership through architecture decisions, code reviews, and mentoring.

Skills

Python

Data Engineering

Problem Solving

Tools

Databricks

PySpark

Spark SQL

Delta Lake

Delta Live Tables

Databricks Auto Loader

Databricks Workflows

Databricks Asset Bundles

AWS

IAM

S3

Unity Catalog

CI/CD

Infrastructure as Code

Job description

BP in Kuala Lumpur is seeking a Senior Data Engineer to own the data domain within an autonomous squad. You will design, build, migrate, operate, and continuously improve scalable data products and pipelines using Databricks, PySpark, Spark SQL, and AWS, ensuring reliability, security and cost efficiency.

You will lead data ingestion, transformation, and storage architectures, enforce data quality and governance, and drive observability and incident resolution.