Databricks Data Engineer: Real-Time Pipelines & Lakehouse

OpenTalent; Singapore; On-site; SGD 90,000 - 140,000; Full time

Job summary

OpenTalent is seeking a skilled Databricks Data Engineer to design, build, and optimize high-throughput batch and real-time data pipelines on the Databricks Lakehouse platform in Singapore. You will implement Delta Lake, Unity Catalog, and advanced CI/CD practices to ensure scalable, secure data infrastructure.

Responsibilities include architecting robust data pipelines with Spark, PySpark, and Spark SQL, ingesting data from APIs, databases and streams, and implementing the Medallion

Qualifications

• Bachelor's degree in CS, data engineering, information systems, or equivalent.
• 3+ years hands-on data engineering with at least 2+ years on Databricks.
• Proficiency in Python (PySpark) and SQL; Spark/Delta Lake expertise.

Responsibilities

• Build and optimize data pipelines using Spark, PySpark, and Spark SQL.
• Ingest data from APIs, databases, cloud storage, and streams.
• Implement Bronze-Silver-Gold with Delta Lake.
• Automate batch and streaming workflows with Databricks Lakeflow or Airflow.
• Enforce access controls and data lineage with Unity Catalog.
• Optimize Spark performance and manage cloud costs.

Skills

Python (PySpark)

SQL

Apache Spark

Delta Lake

Databricks

CI/CD / Testing

Git

Education

Bachelor's degree in Computer Science or related field

Tools

Databricks

Unity Catalog

Apache Airflow

Job description

OpenTalent is seeking a skilled Databricks Data Engineer to design, build, and optimize high-throughput batch and real-time data pipelines on the Databricks Lakehouse platform in Singapore. You will implement Delta Lake, Unity Catalog, and advanced CI/CD practices to ensure scalable, secure data infrastructure.

Responsibilities include architecting robust data pipelines with Spark, PySpark, and Spark SQL, ingesting data from APIs, databases and streams, and implementing the Medallion