Databricks Engineer

K2 Partnering Solutions; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

K2 Partnering Solutions in Singapore is seeking a Databricks Data Engineer to design, build and optimize high-throughput data pipelines on the Databricks Lakehouse platform. You will implement Delta Lake, Unity Catalog and Spark-based processing, collaborate with data science and BI teams, and help enforce CI/CD and testing for reliable data products.

The role emphasizes scalable, secure and cost-aware data infrastructure with hands-on experience in cloud environments (AWS/Azure/GCP) and strong

Qualifications

• Bachelor's degree in Computer Science, Data Engineering, Information Systems, or equivalent practical experience.
• 3+ years of hands-on experience in data engineering, with at least 2+ years actively building on Databricks.
• Strong proficiency in Python (PySpark) and SQL; Scala is a plus.

Responsibilities

• Architecture and deploy robust data pipelines using Apache Spark, PySpark, and Spark SQL to ingest data from diverse sources.
• Implement and manage the Medallion Architecture (Bronze, Silver, Gold) using Delta Lake for clean, analytics-ready datasets.
• Automate batch and streaming workflows using Databricks Lakeflow Jobs, Workflows, or Apache Airflow.
• Set up fine-grained access control, security policies, and data lineage tracking using Unity Catalog.
• Optimize Spark jobs, query performance, and compute cluster configurations to balance performance and cloud costs.
• Partner with Data Scientists, BI Engineers, and product teams to translate requirements into data models and enforce CI/CD testing.

Skills

Data engineering

PySpark

SQL

Python

CI/CD

Git

Spark

Delta Lake

Airflow

Education

Bachelor's degree in Computer Science, Data Engineering, Information Systems, or equivalent practical experience.

Tools

Databricks

Unity Catalog

Delta Lake

Databricks Workflows

Spark SQL

Airflow

dbt

Job description

We are seeking a skilled and experienced Databricks Data Engineer to join our data platform team. In this role, you will design, build, and optimize high-throughput batch and real-time data pipelines on the Databricks Lakehouse platform. You will implement best practices using Delta Lake, Apache Spark, and Unity Catalog to ensure our data infrastructure is scalable, secure, and cost-effective.

Key Responsibilities

• Architecture and deploy robust data pipelines using Apache Spark, PySpark, and Spark SQL to ingest data from diverse sources (APIs, relational databases, cloud storage, event streams).
• Implement and manage the Medallion Architecture (Bronze, Silver, Gold layers) using Delta Lake to deliver clean, transactional, and analytics-ready datasets.
• Automate batch and streaming workflows using Databricks Lakeflow Jobs, Workflows, or Apache Airflow.
• Set up fine-grained access control, security policies, and data lineage tracking using Unity Catalog.
• Optimize Spark jobs, query performance, and compute cluster configurations (autoscaling, caching, partitioning) to maintain performance while minimizing cloud costs.
• Partner with Data Scientists, Business Intelligence Engineers, and product teams to translate business requirements into efficient data models. Enforce unit testing and CI/CD pipelines for data engineering code.

Required Qualifications

• Bachelor's degree in Computer Science, Data Engineering, Information Systems, or equivalent practical experience.
• 3+ years of hands-on experience in data engineering, with at least 2+ years actively building on Databricks.
• Strong proficiency in Python (PySpark) and SQL (Scala is a plus).
• Core Technologies:
• Deep experience with Apache Spark and Delta Lake.
• Hands-on experience with cloud platforms (AWS, Azure, or GCP).
• Familiarity with Unity Catalog for governance and Databricks Workflows for orchestration.
• Strong experience with Git, CI/CD pipelines, and writing testable, modular code.

Preferred Qualifications

• Databricks Certified Data Engineer Associate or Professional.
• Experience with real-time streaming engines (Structured Streaming, Apache Kafka, Kinesis).
• Knowledge of dbt (data build tool) integrated with Databricks.
• Familiarity with machine learning operations (MLflow) and vector databases on Databricks.