Senior Data Engineer: Databricks + AWS Leader

Bpinternational (bpCareers); Kuala Lumpur; Hybrid; MYR 180,000 - 260,000; Full time

Benefits offered by this job

Life insurance

Health insurance

Career development

Learning opportunities

Diversity culture

Job summary

bp is seeking a Senior Data Engineer to own end-to-end data domain ownership within the squad. You will design, build, migrate, operate, and continuously improve data products and pipelines to be scalable, reliable, secure, observable, and cost-effective.

The role requires 8+ years in data/software engineering, strong Python and PySpark experience, and expertise in Databricks, AWS, and governance tools. This hybrid position offers opportunities for leadership and technical mentorship.

Qualifications

• 8+ years of experience in Data Engineering, Software Engineering, or related technical disciplines.
• 5+ years hands-on Python experience and 3+ years working with PySpark and distributed data processing.
• Strong hands-on expertise in Databricks, PySpark, Spark SQL, SQL, Delta Lake, and data modelling.
• Experience designing and operating enterprise-scale data ingestion, transformation, migration, and processing frameworks.
• Hands-on experience with Databricks Auto Loader, Delta Live Tables (DLT), Workflows, and Databricks Asset Bundles (DAB).
• Strong AWS experience, including IAM, S3 and integration with AWS cloud services and infrastructure.
• Experience with Unity Catalog, including data governance, access control, metadata, and lineage.
• Experience with CI/CD, Infrastructure as Code, automated testing, Git, and deployment automation.
• Strong understanding of observability, data quality, performance optimisation, cost monitoring, reliability, and production support.
• Good understanding of information security, data governance, privacy, resilience, and operational best practices.
• Strong technical leadership, problem-solving, communication, stakeholder management, and mentoring skills.

Responsibilities

• Lead the design, development, migration, and operation of scalable data pipelines and data products using Databricks, PySpark, Spark SQL, and AWS.
• Own the squad’s data engineering domain, providing technical direction and ensuring solutions are reliable, supportable, secure, and aligned with business needs.
• Design and implement scalable data ingestion and migration frameworks using Databricks Auto Loader and Delta Live Tables (DLT).
• Develop and maintain data models, transformation pipelines, and storage solutions with strong focus on data quality, integrity, performance, and scalability.
• Build and maintain pipeline orchestration and deployment capabilities using Databricks Workflows, Databricks Asset Bundles (DAB), CI/CD, and Infrastructure as Code.
• Integrate data platforms with the AWS ecosystem, including IAM, storage, security, networking, and supporting cloud services.
• Implement effective governance, security, lineage, and access controls using Unity Catalog and enterprise standards.
• Establish observability, monitoring, alerting, and cost optimisation across data pipelines and Databricks workloads.
• Drive operational excellence through incident resolution, root cause analysis, automation, runbooks, and continuous reliability improvements.
• Provide technical leadership through architecture decisions, code reviews, engineering standards, mentoring, and knowledge sharing.

Skills

Python

Databricks

Spark SQL

SQL

Delta Lake

AWS

Unity Catalog

CI/CD

IaC

Observability

Data governance

Data quality

Data modeling

Security

Mentoring

Tools

Databricks Auto Loader

Delta Live Tables

Databricks Workflows

Databricks Asset Bundles

IAM

S3

Git

Job description

bp is seeking a Senior Data Engineer to own end-to-end data domain ownership within the squad. You will design, build, migrate, operate, and continuously improve data products and pipelines to be scalable, reliable, secure, observable, and cost-effective.

The role requires 8+ years in data/software engineering, strong Python and PySpark experience, and expertise in Databricks, AWS, and governance tools. This hybrid position offers opportunities for leadership and technical mentorship.