Databricks Data Engineer: Spark, Delta Lake & Pipelines

K2 PARTNERING SOLUTIONS PTE. LTD.; Singapore; On-site; SGD 90,000 - 130,000; Full time

Job summary

K2 PARTNERING SOLUTIONS PTE. LTD. is seeking a Databricks Data Engineer to design, build, and optimize high-throughput batch and real-time data pipelines on the Databricks Lakehouse platform in Singapore.

You will implement Delta Lake, Unity Catalog, and Spark-based architectures to ensure scalable and secure data infrastructure. You will work with Data Scientists and BI teams to translate business requirements into efficient data models, orchestrate workflows with Airflow, and enforce robust

Qualifications

• Bachelor's degree in CS, Data Engineering, Information Systems, or equivalent practical experience.
• 3+ years in data engineering, with at least 2+ years on Databricks.
• Strong Python (PySpark) and SQL skills; Scala is a plus.

Responsibilities

• Architect and deploy robust data pipelines using Spark, PySpark, and Spark SQL from diverse sources.
• Implement and manage Medallion Architecture (Bronze-Silver-Gold) with Delta Lake for clean datasets.
• Automate batch and streaming workflows using Databricks Lakeflow Jobs or Airflow.
• Set up fine-grained access control and data lineage with Unity Catalog.
• Optimize Spark jobs and cluster configs to balance performance and cloud costs.
• Collaborate with Data Scientists and BI Engineers to translate requirements into data models and enforce CI/CD.

Skills

Python

PySpark

SQL

Spark

Delta Lake

CI/CD

Git

Education

Bachelor's degree

Tools

Databricks

Unity Catalog

Airflow

Job description

K2 PARTNERING SOLUTIONS PTE. LTD. is seeking a Databricks Data Engineer to design, build, and optimize high-throughput batch and real-time data pipelines on the Databricks Lakehouse platform in Singapore.

You will implement Delta Lake, Unity Catalog, and Spark-based architectures to ensure scalable and secure data infrastructure. You will work with Data Scientists and BI teams to translate business requirements into efficient data models, orchestrate workflows with Airflow, and enforce robust