Databricks Engineer

ESPIRE INFOLABS (SINGAPORE) PTE. LTD.; Singapore; On-site; SGD 90,000 - 130,000; Full time

Job summary

ESPIRE INFOLABS (SINGAPORE) PTE. LTD. is seeking a hands-on Databricks/Data Engineer to design, develop and maintain scalable data pipelines on the Databricks Lakehouse Platform.

You will leverage Spark, PySpark, Python, SQL and Delta Lake to deliver reliable, high-performance data solutions. The role involves working with Azure, AWS or GCP data services, building batch and near-real-time processing, ensuring data quality and governance, and collaborating with architects, data scientists and BI

Qualifications

• 3–5 years of experience in data engineering.
• Hands-on Databricks expertise.
• Strong Spark/PySpark, Python, and SQL skills.
• Experience with Delta Lake and cloud data platforms.

Responsibilities

• Design, develop, and maintain scalable data pipelines.
• Build ETL/ELT pipelines using PySpark, Python, and SQL.
• Create and manage Delta Lake tables and workflows.
• Work with Databricks Lakehouse architecture and components.
• Ingest and transform data from databases, APIs, files.
• Implement data cleansing, validation, and quality checks.
• Optimize Spark jobs for performance and cost.
• Collaborate with data architects, data scientists, BI teams.

Skills

Databricks

Apache Spark

PySpark

Python

SQL

Delta Lake

ETL/ELT

Azure

Data Warehousing

Data Modelling

Git

CI/CD

Tools

Azure Data Factory

Terraform

Databricks Workflows

Unity Catalog

Power BI

Job description

About the Role

We are looking for a skilled Databricks Engineer todesign, develop, and maintain scalable data engineering solutions using the DatabricksLakehouse Platform.

The ideal candidate will have strong hands‑on experiencewith Databricks, Apache Spark, Python, SQL, Delta Lake, and cloud dataplatforms, with the ability to build reliable and high‑performance datapipelines.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
• Develop ETL/ELT pipelines using PySpark, Python, and SQL.
• Build and maintain Delta Lake tables and data processing workflows.
• Work with Databricks Lakehouse architecture and related data engineering components.
• Develop batch and, where required, near-real-time data processing solutions.
• Ingest and transform data from databases, APIs, files, and other data sources.
• Implement data cleansing, transformation, validation, and quality checks.
• Optimise Spark jobs and Databricks workloads for performance and cost efficiency.
• Work with cloud storage and data services across Azure, AWS, or GCP.
• Implement data security, access controls, and governance within the data platform.
• Collaborate with Data Architects, Data Scientists, BI Developers, and business stakeholders.
• Troubleshoot data pipeline failures and resolve performance and data-quality issues.
• Develop and maintain technical documentation for data pipelines and solutions.
• Participate in code reviews, testing, deployment, and production support.
• Follow Agile development practices and contribute to continuous improvement.

Required Skills & Experience

• 3–5 years of experience in Data Engineering.
• Strong hands‑on experience with Databricks.
• Strong knowledge of:
• Apache Spark / PySpark
• Python
• SQL
• Delta Lake
• ETL/ELT concepts
• Experience developing and managing data pipelines.
• Good understanding of data warehousing and data modelling concepts.
• Experience working with cloud platforms, preferably Microsoft Azure.
• Experience with cloud storage such as Azure Data Lake Storage (ADLS), Amazon S3, or Google Cloud Storage.
• Experience with relational and/or NoSQL databases.
• Good understanding of data quality, validation, and governance.
• Familiarity with Git and CI/CD practices.
• Strong troubleshooting and analytical skills.

Good to Have

• Databricks Certified Data Engineer Associate/Professional certification.
• Experience with Azure Data Factory.
• Experience with Microsoft Fabric.
• Knowledge of Unity Catalog and Databricks governance.
• Experience with Databricks Workflows and job orchestration.
• Experience with streaming technologies such as Kafka or Structured Streaming.
• Experience with Power BI or other BI platforms.
• Exposure to Machine Learning workflows on Databricks.
• Experience with Terraform or Infrastructure as Code.
• Knowledge of DevOps and CI/CD pipelines.

Candidate Profile

The ideal candidate should be a hands‑on Databricks/Data Engineer capable of independently developing data pipelines, troubleshooting production issues, optimising Spark workloads, and collaborating with technical and business teams.