Senior Data Engineer: Real-Time Pipelines & Lakehouse

Agensi Pekerjaan Penta Consultancy Sdn. Bhd; Kuala Lumpur; Sur place; MYR 180 000 - 240 000; Plein temps

Résumé du poste

Agensi Pekerjaan Penta Consultancy Sdn. Bhd is seeking an experienced Data Engineer to design, develop, and maintain scalable data pipelines using Databricks, PySpark and Delta Lake in a cloud environment.

You will build ETL/ELT processes, manage data governance, and optimize performance across warehouses and lakehouses. The role requires deep experience with Unity Catalog, RBAC/ABAC, Delta Sharing, and CI/CD via GitHub Actions.

Qualifications

• Bachelor's degree in CS/IT/Engineering/Data Science or related field.
• 5+ years in Data Engineering, Data Warehousing or Big Data.
• Hands-on with Databricks/Azure Databricks, PySpark, Delta Lake/Delta Tables, Databricks Workflows.
• Knowledge of Unity Catalog, RBAC/ABAC, Delta Sharing, data governance and security.
• Experience with Databricks Genie/Genie Spaces preferred.
• Strong SQL and data warehouse performance optimization.
• Experience with cloud platforms (Azure/AWS/GCP).
• Infrastructure-as-Code knowledge (Terraform) is a plus.

Responsabilités

• Design, develop, and maintain scalable data pipelines and data solutions using Databricks and PySpark.
• Build and optimize ETL/ELT solutions for batch and near real-time processing.
• Design and maintain Data Warehouse and Lakehouse solutions (Delta Lake) including Delta Tables.
• Develop dimensional models (Fact/Dimension, Star/Snowflake, SCD 1 & 2).
• Implement and maintain data governance, RBAC/ABAC, auditing and data lineage.
• Configure Delta Sharing for secure data collaboration.
• Support Databricks Genie/Genie Spaces adoption and administration.
• Perf tuning for PySpark, SQL, and data warehouses; optimize clusters/workloads.
• Develop CI/CD pipelines and automate deployments with GitHub Actions.
• Collaborate with business users and engineers to translate requirements into scalable data solutions.
• Provide technical guidance and mentorship to juniors.
• Continuously improve platform performance, security, governance, and cost efficiency.

Connaissances

Analytical thinking

Problem-solving

Communication

Stakeholder management

Formation

Bachelor's Degree in Computer Science / IT / Engineering / Data Science

Outils

Databricks

Azure Databricks

PySpark

Delta Lake

Delta Tables

Databricks Workflows

Unity Catalog

RBAC

ABAC

Delta Sharing

GitHub Actions

Terraform

Description du poste

Agensi Pekerjaan Penta Consultancy Sdn. Bhd is seeking an experienced Data Engineer to design, develop, and maintain scalable data pipelines using Databricks, PySpark and Delta Lake in a cloud environment.

You will build ETL/ELT processes, manage data governance, and optimize performance across warehouses and lakehouses. The role requires deep experience with Unity Catalog, RBAC/ABAC, Delta Sharing, and CI/CD via GitHub Actions.