Senior Data Engineer - Databricks & Lakehouse Architect

Michael Page; Kuala Selangor; On-site; MYR 120,000 - 180,000; Full time

Benefits offered by this job

Competitive salary package

Exposure to enterprise Databricks

Job summary

Michael Page Malaysia is recruiting a Senior Data Engineer to design and maintain scalable data pipelines and data products using Databricks and PySpark. The role develops enterprise Data Warehouse and Lakehouse solutions for analytics, reporting, and AI/ML use cases.

The candidate will create dimensional models including Fact and Dimension tables, Star and Snowflake schemas, and Slowly Changing Dimensions, while ensuring data quality, security, and performance at scale.

Qualifications

• Bachelor's degree in a technical field required.
• 8+ years in Data Engineering, Data Warehousing or big data environments.
• Strong PySpark, Databricks, and Lakehouse experience.
• Proven ability to design enterprise DW and dimensional models.
• Experience implementing CI/CD pipelines and DevOps best practices.

Responsibilities

• Design, develop, and maintain scalable data pipelines and data products using Databricks and PySpark.
• Build and optimize ETL/ELT frameworks for batch and near real-time processing.
• Develop enterprise Data Warehouse and Lakehouse solutions for analytics and AI/ML use cases.
• Create dimensional data models including Fact/Dimension tables, Star/Snowflake schemas, and SCDs.
• Ensure data quality, security, scalability, and platform performance.
• Implement Databricks governance solutions including Unity Catalog, RBAC, ABAC, data lineage, and auditing.
• Optimize platform performance via PySpark tuning, workload management, and cost efficiency.
• Design and implement CI/CD pipelines, deployment automation, and DevOps for Databricks.
• Collaborate with stakeholders and technology teams to deliver scalable data solutions.
• Provide technical guidance and participate in production support and incident management.

Skills

Databricks

PySpark

ETL/ELT

Data Warehouse

Lakehouse

Dimensional data models

CI/CD pipelines

DevOps

Data governance

Security & RBAC/ABAC

Education

Bachelor's Degree in Computer Science, IT, Engineering, Data Science, or related

Tools

Unity Catalog

RBAC/ABAC frameworks

Data lineage tooling

GitHub integration

Job description

Michael Page Malaysia is recruiting a Senior Data Engineer to design and maintain scalable data pipelines and data products using Databricks and PySpark. The role develops enterprise Data Warehouse and Lakehouse solutions for analytics, reporting, and AI/ML use cases.

The candidate will create dimensional models including Fact and Dimension tables, Star and Snowflake schemas, and Slowly Changing Dimensions, while ensuring data quality, security, and performance at scale.