Databricks Engineer - K2 PARTNERING SOLUTIONS PTE. LTD.

OpenTalent; Singapore; On-site; SGD 90,000 - 140,000; Full time

Job summary

OpenTalent is seeking a skilled Databricks Data Engineer to design, build, and optimize high-throughput batch and real-time data pipelines on the Databricks Lakehouse platform in Singapore. You will implement Delta Lake, Unity Catalog, and advanced CI/CD practices to ensure scalable, secure data infrastructure.

Responsibilities include architecting robust data pipelines with Spark, PySpark, and Spark SQL, ingesting data from APIs, databases and streams, and implementing the Medallion

Qualifications

• Bachelor's degree in CS, data engineering, information systems, or equivalent.
• 3+ years hands-on data engineering with at least 2+ years on Databricks.
• Proficiency in Python (PySpark) and SQL; Spark/Delta Lake expertise.

Responsibilities

• Build and optimize data pipelines using Spark, PySpark, and Spark SQL.
• Ingest data from APIs, databases, cloud storage, and streams.
• Implement Bronze-Silver-Gold with Delta Lake.
• Automate batch and streaming workflows with Databricks Lakeflow or Airflow.
• Enforce access controls and data lineage with Unity Catalog.
• Optimize Spark performance and manage cloud costs.

Skills

Python (PySpark)

SQL

Apache Spark

Delta Lake

Databricks

CI/CD / Testing

Git

Education

Bachelor's degree in Computer Science or related field

Tools

Databricks

Unity Catalog

Apache Airflow

Job description

Role Overview

We are seeking a skilled and experienced Databricks Data Engineer to join our data platform team. In this role, you will design, build, and optimize high-throughput batch and real-time data pipelines on the Databricks Lakehouse platform. You will implement best practices using Delta Lake, Apache Spark, and Unity Catalog to ensure our data infrastructure is scalable, secure, and cost-effective.

Key Responsibilities

• Architecture and deploy robust data pipelines using Apache Spark, PySpark, and Spark SQL to ingest data from diverse sources (APIs, relational databases, cloud storage, event streams).
• Implement and manage the Medallion Architecture (Bronze, Silver, Gold layers) using Delta Lake to deliver clean, transactional, and analytics-ready datasets.
• Automate batch and streaming workflows using Databricks Lakeflow Jobs, Workflows, or Apache Airflow.
• Set up fine-grained access control, security policies, and data lineage tracking using Unity Catalog .
• Optimize Spark jobs, query performance, and compute cluster configurations (autoscaling, caching, partitioning) to maintain performance while minimizing cloud costs.
• Partner with Data Scientists, Business Intelligence Engineers, and product teams to translate business requirements into efficient data models. Enforce unit testing and CI/CD pipelines for data engineering code.

Required Qualifications

• Bachelor's degree in Computer Science, Data Engineering, Information Systems, or equivalent practical experience.
• 3+ years of hands-on experience in data engineering, with at least 2+ years actively building on Databricks .
• Strong proficiency in Python (PySpark) and SQL (Scala is a plus).
• Deep experience with Apache Spark and Delta Lake .
• Hands-on experience with cloud platforms ( AWS , Azure , or GCP ).
• Familiarity with Unity Catalog for governance and Databricks Workflows for orchestration.
• Strong experience with Git, CI/CD pipelines, and writing testable, modular code.

Preferred Qualifications

• Databricks Certified Data Engineer Associate or Professional.
• Experience with real-time streaming engines (Structured Streaming, Apache Kafka, Kinesis).
• Knowledge of dbt (data build tool) integrated with Databricks.
• Familiarity with machine learning operations (MLflow) and vector databases on Databricks.