Databricks Data Engineer: Scale Data Pipelines on Lakehouse

K2 Partnering Solutions Pte Ltd; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

K2 Partnering Solutions Pte Ltd is seeking a Databricks Data Engineer to design, build, and optimize data pipelines on the Databricks Lakehouse platform. You will implement Delta Lake, Apache Spark, and Unity Catalog to ensure scalable, secure, and cost-effective data infrastructure.

Collaborating with data scientists and BI engineers, you will translate business requirements into efficient data models, and advance batch and streaming workflows using Databricks Workflows or Apache Airflow.

Qualifications

• Bachelor's degree in Computer Science, Data Engineering, Information Systems, or equivalent practical experience.
• 3+ years of hands-on experience in data engineering, with at least 2+ years actively building on Databricks.
• Strong proficiency in Python (PySpark) and SQL (Scala is a plus).

Responsibilities

• Architecture and deploy robust data pipelines using Apache Spark, PySpark, and Spark SQL to ingest data from diverse sources (APIs, relational databases, cloud storage, event streams).
• Implement and manage the Medallion Architecture (Bronze, Silver, Gold layers) using Delta Lake to deliver clean, transactional, and analytics-ready datasets.
• Automate batch and streaming workflows using Databricks Lakeflow Jobs, Workflows, or Apache Airflow.
• Set up fine-grained access control, security policies, and data lineage tracking using Unity Catalog.
• Optimize Spark jobs, query performance, and compute cluster configurations (autoscaling, caching, partitioning) to maintain performance while minimizing cloud costs.
• Partner with Data Scientists, Business Intelligence Engineers, and product teams to translate business requirements into efficient data models. Enforce unit testing and CI/CD pipelines for data engineering code.

Skills

Python (PySpark)

SQL

Spark

CI/CD

Git

Education

Bachelor's degree in Computer Science

Tools

Databricks

Delta Lake

Unity Catalog

Spark SQL

Airflow

Job description

K2 Partnering Solutions Pte Ltd is seeking a Databricks Data Engineer to design, build, and optimize data pipelines on the Databricks Lakehouse platform. You will implement Delta Lake, Apache Spark, and Unity Catalog to ensure scalable, secure, and cost-effective data infrastructure.

Collaborating with data scientists and BI engineers, you will translate business requirements into efficient data models, and advance batch and streaming workflows using Databricks Workflows or Apache Airflow.