Databricks Engineer

K2 PARTNERING SOLUTIONS PTE. LTD.; Singapore; On-site; SGD 90,000 - 130,000; Full time

Job summary

K2 PARTNERING SOLUTIONS PTE. LTD. is seeking a Databricks Data Engineer to design, build, and optimize high-throughput batch and real-time data pipelines on the Databricks Lakehouse platform in Singapore.

You will implement Delta Lake, Unity Catalog, and Spark-based architectures to ensure scalable and secure data infrastructure. You will work with Data Scientists and BI teams to translate business requirements into efficient data models, orchestrate workflows with Airflow, and enforce robust

Qualifications

• Bachelor's degree in CS, Data Engineering, Information Systems, or equivalent practical experience.
• 3+ years in data engineering, with at least 2+ years on Databricks.
• Strong Python (PySpark) and SQL skills; Scala is a plus.

Responsibilities

• Architect and deploy robust data pipelines using Spark, PySpark, and Spark SQL from diverse sources.
• Implement and manage Medallion Architecture (Bronze-Silver-Gold) with Delta Lake for clean datasets.
• Automate batch and streaming workflows using Databricks Lakeflow Jobs or Airflow.
• Set up fine-grained access control and data lineage with Unity Catalog.
• Optimize Spark jobs and cluster configs to balance performance and cloud costs.
• Collaborate with Data Scientists and BI Engineers to translate requirements into data models and enforce CI/CD.

Skills

Python

PySpark

SQL

Spark

Delta Lake

CI/CD

Git

Education

Bachelor's degree

Tools

Databricks

Unity Catalog

Airflow

Job description

Role Overview

We are seeking a skilled and experienced Databricks Data Engineer to join our data platform team. In this role, you will design, build, and optimize high-throughput batch and real-time data pipelines on the Databricks Lakehouse platform. You will implement best practices using Delta Lake, Apache Spark, and Unity Catalog to ensure our data infrastructure is scalable, secure, and cost-effective.

Key Responsibilities

• Architecture and deploy robust data pipelines using Apache Spark, PySpark, and Spark SQL to ingest data from diverse sources (APIs, relational databases, cloud storage, event streams).
• Implement and manage the Medallion Architecture (Bronze, Silver, Gold layers) using Delta Lake to deliver clean, transactional, and analytics-ready datasets.
• Automate batch and streaming workflows using Databricks Lakeflow Jobs, Workflows, or Apache Airflow.
• Set up fine-grained access control, security policies, and data lineage tracking using Unity Catalog .
• Optimize Spark jobs, query performance, and compute cluster configurations (autoscaling, caching, partitioning) to maintain performance while minimizing cloud costs.
• Partner with Data Scientists, Business Intelligence Engineers, and product teams to translate business requirements into efficient data models. Enforce unit testing and CI/CD pipelines for data engineering code.

Required Qualifications

Bachelor's degree in Computer Science, Data Engineering, Information Systems, or equivalent practical experience.

3+ years of hands-on experience in data engineering, with at least 2+ years actively building on Databricks .

Strong proficiency in Python (PySpark) and SQL (Scala is a plus).

Core Technologies:

• Deep experience with Apache Spark and Delta Lake .
• Hands-on experience with cloud platforms ( AWS , Azure , or GCP ).
• Familiarity with Unity Catalog for governance and Databricks Workflows for orchestration.
• Strong experience with Git, CI/CD pipelines, and writing testable, modular code.

Preferred Qualifications

• Databricks Certified Data Engineer Associate or Professional.
• Experience with real-time streaming engines (Structured Streaming, Apache Kafka, Kinesis).
• Knowledge of dbt (data build tool) integrated with Databricks.
• Familiarity with machine learning operations (MLflow) and vector databases on Databricks.