Senior Data Engineer

Bpinternational (bpCareers); Kuala Lumpur; Hybrid; MYR 180,000 - 260,000; Full time

Benefits offered by this job

Life insurance

Health insurance

Career development

Learning opportunities

Diversity culture

Job summary

bp is seeking a Senior Data Engineer to own end-to-end data domain ownership within the squad. You will design, build, migrate, operate, and continuously improve data products and pipelines to be scalable, reliable, secure, observable, and cost-effective.

The role requires 8+ years in data/software engineering, strong Python and PySpark experience, and expertise in Databricks, AWS, and governance tools. This hybrid position offers opportunities for leadership and technical mentorship.

Qualifications

• 8+ years of experience in Data Engineering, Software Engineering, or related technical disciplines.
• 5+ years hands-on Python experience and 3+ years working with PySpark and distributed data processing.
• Strong hands-on expertise in Databricks, PySpark, Spark SQL, SQL, Delta Lake, and data modelling.
• Experience designing and operating enterprise-scale data ingestion, transformation, migration, and processing frameworks.
• Hands-on experience with Databricks Auto Loader, Delta Live Tables (DLT), Workflows, and Databricks Asset Bundles (DAB).
• Strong AWS experience, including IAM, S3 and integration with AWS cloud services and infrastructure.
• Experience with Unity Catalog, including data governance, access control, metadata, and lineage.
• Experience with CI/CD, Infrastructure as Code, automated testing, Git, and deployment automation.
• Strong understanding of observability, data quality, performance optimisation, cost monitoring, reliability, and production support.
• Good understanding of information security, data governance, privacy, resilience, and operational best practices.
• Strong technical leadership, problem-solving, communication, stakeholder management, and mentoring skills.

Responsibilities

• Lead the design, development, migration, and operation of scalable data pipelines and data products using Databricks, PySpark, Spark SQL, and AWS.
• Own the squad’s data engineering domain, providing technical direction and ensuring solutions are reliable, supportable, secure, and aligned with business needs.
• Design and implement scalable data ingestion and migration frameworks using Databricks Auto Loader and Delta Live Tables (DLT).
• Develop and maintain data models, transformation pipelines, and storage solutions with strong focus on data quality, integrity, performance, and scalability.
• Build and maintain pipeline orchestration and deployment capabilities using Databricks Workflows, Databricks Asset Bundles (DAB), CI/CD, and Infrastructure as Code.
• Integrate data platforms with the AWS ecosystem, including IAM, storage, security, networking, and supporting cloud services.
• Implement effective governance, security, lineage, and access controls using Unity Catalog and enterprise standards.
• Establish observability, monitoring, alerting, and cost optimisation across data pipelines and Databricks workloads.
• Drive operational excellence through incident resolution, root cause analysis, automation, runbooks, and continuous reliability improvements.
• Provide technical leadership through architecture decisions, code reviews, engineering standards, mentoring, and knowledge sharing.

Skills

Python

Databricks

Spark SQL

SQL

Delta Lake

AWS

Unity Catalog

CI/CD

IaC

Observability

Data governance

Data quality

Data modeling

Security

Mentoring

Tools

Databricks Auto Loader

Delta Live Tables

Databricks Workflows

Databricks Asset Bundles

IAM

S3

Git

Job description

Entity:

Technology

Job Family Group:

IT&S Group

Job Description:

Let me tell you about the role

As the primary Data Engineer within the squad, this role provides end-to-end technical ownership of the data domain. The Senior Data Engineer will design, build, migrate, operate, and continuously improve data products and pipelines, ensuring they are scalable, reliable, secure, observable, and cost-effective.

Key Responsibilities

• Lead the design, development, migration, and operation of scalable data pipelines and data products using Databricks, PySpark, Spark SQL, and AWS.
• Own the squad’s data engineering domain, providing technical direction and ensuring solutions are reliable, supportable, secure, and aligned with business needs.
• Design and implement scalable data ingestion and migration frameworks using technologies such as Databricks Auto Loader and Delta Live Tables (DLT).
• Develop and maintain data models, transformation pipelines, and storage solutions with strong focus on data quality, integrity, performance, and scalability.
• Build and maintain pipeline orchestration and deployment capabilities using Databricks Workflows, Databricks Asset Bundles (DAB), CI/CD, and Infrastructure as Code.
• Integrate data platforms with the AWS ecosystem, including IAM, storage, security, networking, and supporting cloud services.
• Implement effective governance, security, lineage, and access controls using Unity Catalog and enterprise standards.
• Establish observability, monitoring, alerting, and cost optimisation across data pipelines and Databricks workloads.
• Drive operational excellence through incident resolution, root cause analysis, automation, runbooks, and continuous reliability improvements.
• Provide technical leadership through architecture decisions, code reviews, engineering standards, mentoring, and knowledge sharing.

What you will need to be successful (experience and qualifications)

• 8+ years of experience in Data Engineering, Software Engineering, or related technical disciplines.
• 5+ years of hands‑on Python experience and 3+ years working with PySpark and distributed data processing.
• Strong hands‑on expertise in Databricks, PySpark, Spark SQL, SQL, Delta Lake, and data modelling.
• Experience designing and operating enterprise-scale data ingestion, transformation, migration, and processing frameworks.
• Hands‑on experience with Databricks Auto Loader, Delta Live Tables (DLT), Workflows, and Databricks Asset Bundles (DAB).
• Strong AWS experience, including IAM, S3 and integration with AWS cloud services and infrastructure.
• Experience with Unity Catalog, including data governance, access control, metadata, and lineage.
• Experience with CI/CD, Infrastructure as Code, automated testing, Git, and deployment automation.
• Strong understanding of observability, data quality, performance optimisation, cost monitoring, reliability, and production support.
• Good understanding of information security, data governance, privacy, resilience, and operational best practices.
• Strong technical leadership, problem-solving, communication, stakeholder management, and mentoring skills.

About bp

At bp, we provide the following environment and benefits to you:

• A company culture where we respect our diverse and unified teams, where we are proud of our achievements and where fun and the attitude of giving back to our environment are highly valued.
• Possibility to join our social communities and networks
• Learning opportunities and other development opportunities to craft your career path
• Life and health insurance, medical care package

And many other benefits. We are an equal opportunity employer and value diversity at our company. We do not discriminate based on race, religion, colour, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status. We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, perform crucial job functions, and receive other benefits and privileges of employment.

Travel Requirement

No travel is expected with this role

Relocation Assistance:

This role is not eligible for relocation

Remote Type:

This position is a hybrid of office/remote working

Skills:

Cloud Platforms, Cloud Platforms, Collaboration, Communication, Configuration management and release, Continuous deployment and release, Creating a high performing team, Database Design, Digital Project Management, Documentation and knowledge sharing, Emerging technology monitoring, Facilitation, Information Security, Mentoring, Metrics definition and instrumentation, NoSql data modelling, Problem Solving, Relational Data Modeling, Risk Management, Scripting, Secure development, Service operations and resiliency, Software Design and Development, Solution Architecture, Source control and code management {+ 5 more}

.

Legal Disclaimer:

We are an equal opportunity employer. We do not discriminate on the basis of protected characteristics like race, religion, color, sex, national origin, sexual orientation, veteran status or disability status.Individuals with an accessibility need may request an adjustment/accommodation related to bp’s recruiting process (e.g., accessing the job application, completing required assessments, participating in telephone screenings or interviews, etc.). If you would like to request an adjustment/accommodation related to the recruitment process,pleasecontact us.
If you are selected for a position and depending upon your role, your employment may be contingent upon adherence to local policy. This may include pre-placement drug screening, medical review of physical fitness for the role, and background checks.