Senior Data Engineer

Agensi Pekerjaan Penta Consultancy Sdn. Bhd; Kuala Lumpur; On-site; MYR 180,000 - 240,000; Full time

Job summary

Agensi Pekerjaan Penta Consultancy Sdn. Bhd is seeking an experienced Data Engineer to design, develop, and maintain scalable data pipelines using Databricks, PySpark and Delta Lake in a cloud environment.

You will build ETL/ELT processes, manage data governance, and optimize performance across warehouses and lakehouses. The role requires deep experience with Unity Catalog, RBAC/ABAC, Delta Sharing, and CI/CD via GitHub Actions.

Qualifications

• Bachelor's degree in CS/IT/Engineering/Data Science or related field.
• 5+ years in Data Engineering, Data Warehousing or Big Data.
• Hands-on with Databricks/Azure Databricks, PySpark, Delta Lake/Delta Tables, Databricks Workflows.
• Knowledge of Unity Catalog, RBAC/ABAC, Delta Sharing, data governance and security.
• Experience with Databricks Genie/Genie Spaces preferred.
• Strong SQL and data warehouse performance optimization.
• Experience with cloud platforms (Azure/AWS/GCP).
• Infrastructure-as-Code knowledge (Terraform) is a plus.

Responsibilities

• Design, develop, and maintain scalable data pipelines and data solutions using Databricks and PySpark.
• Build and optimize ETL/ELT solutions for batch and near real-time processing.
• Design and maintain Data Warehouse and Lakehouse solutions (Delta Lake) including Delta Tables.
• Develop dimensional models (Fact/Dimension, Star/Snowflake, SCD 1 & 2).
• Implement and maintain data governance, RBAC/ABAC, auditing and data lineage.
• Configure Delta Sharing for secure data collaboration.
• Support Databricks Genie/Genie Spaces adoption and administration.
• Perf tuning for PySpark, SQL, and data warehouses; optimize clusters/workloads.
• Develop CI/CD pipelines and automate deployments with GitHub Actions.
• Collaborate with business users and engineers to translate requirements into scalable data solutions.
• Provide technical guidance and mentorship to juniors.
• Continuously improve platform performance, security, governance, and cost efficiency.

Skills

Analytical thinking

Problem-solving

Communication

Stakeholder management

Education

Bachelor's Degree in Computer Science / IT / Engineering / Data Science

Tools

Databricks

Azure Databricks

PySpark

Delta Lake

Delta Tables

Databricks Workflows

Unity Catalog

RBAC

ABAC

Delta Sharing

GitHub Actions

Terraform

Job description

• Design, develop, and maintain scalable data pipelines and data solutions using Databricks and PySpark.
• Build and optimize ETL/ELT solutions for batch and near real-time data processing.
• Design and maintain enterprise Data Warehouse and Lakehouse solutions, including Delta Lake and Delta Tables.
• Develop dimensional data models using Fact/Dimension tables, Star Schema, Snowflake Schema, and SCD Type 1 & 2.
• Implement and maintain data governance and security using Unity Catalog, RBAC, ABAC, data lineage, auditing, and fine-grained access controls.
• Configure and manage Delta Sharing for secure internal and external data collaboration.
• Support Databricks Genie and Genie Spaces adoption and administration.
• Perform PySpark, SQL, Data Warehouse and Lakehouse performance tuning, including query, cluster and workload optimization.
• Develop and maintain CI/CD pipelines and automate deployment, testing and release processes using GitHub, GitHub Actions or equivalent tools.
• Collaborate with business users, data analysts, architects and technology teams to translate business requirements into scalable data solutions.
• Provide technical guidance and mentorship to junior team members where required.
• Continuously improve data platform performance, scalability, security, governance and cost efficiency.
  

Job Requirements:

• Bachelor's Degree in Computer Science, Information Technology, Engineering, Data Science or a related field.
• 5 years and aboveof experience in Data Engineering, Data Warehousing or Big Data technologies.
• Strong hands-on experience with Databricks, Azure Databricks, PySpark, Delta Lake, Delta Tables and Databricks Workflows.
• Strong knowledge of Unity Catalog, RBAC, ABAC, Delta Sharing, data governance and data security.
• Experience with Databricks Genie / Genie Spaces is preferred.
• Strong SQL skills with experience in query and data warehouse performance optimization.
• Solid experience in Data Warehouse architecture, Lakehouse architecture and Medallion design patterns.
• Strong knowledge of dimensional data modelling, including Fact/Dimension tables, Star Schema, Snowflake Schema and SCD.
• Experience with GitHub, Git-based workflows and CI/CD, preferably using GitHub Actions.
• Experience with at least one major cloud platform (Azure, AWS or GCP).
• Knowledge of Infrastructure-as-Code, preferably Terraform, data observability and monitoring is an advantage.
• Databricks certification is an added advantage.
• Strong analytical, problem-solving and troubleshooting skills.
• Good communication and stakeholder management skills, with the ability to work independently and across business and technical teams.