Databricks Engineer — 12-Month Data Pipeline Contract

PERSOL Outsourcing; Singapore; On-site; SGD 90,000 - 150,000; Full time

Job summary

PERSOL Outsourcing is seeking a skilled Data Engineer to design and operate batch and streaming pipelines using Databricks and AWS. You will work across Bronze/Silver/Gold layers, building scalable data processing with PySpark and Delta Lake.

Join a fast-paced team, implement robust data ingestion from APIs and files, ensure data quality, performance, and reliability, and contribute to CI/CD and production deployments.

Qualifications

• 3+ years in data engineering or enterprise data-platform delivery.
• Hands-on with Databricks, Apache Spark, PySpark, and SQL.
• Experience with Delta Lake and Bronze/Silver/Gold data architecture.
• Production data pipelines with logging, error handling, and recovery.
• Experience with AWS services such as S3, IAM, Glue, Lambda, Kinesis.
• Strong data modelling, data quality, and performance optimisation.
• CI/CD and infrastructure/configuration automation using Git-based tools.
• Experience with APIs, XML/JSON ingestion.

Responsibilities

• Design and build batch and streaming data pipelines using Databricks and AWS.
• Develop data processing across Bronze, Silver, and Gold layers using SQL, Python/PySpark, and Delta Lake.
• Implement ingestion from APIs, files, databases, object storage, or other enterprise data sources.
• Develop data transformations, dimensional or curated datasets, and reusable data-engineering components.
• Implement data validation, reconciliation, error handling, logging, rerun/recovery, and monitoring.
• Optimise Spark jobs, clusters, partitioning, and storage for performance and cost.
• Support CI/CD, environment promotion, SIT/UAT, production deployment, documentation, and operational handover.
• Collaborate with platform, application, BI, and data-governance teams.

Skills

Databricks

Apache Spark

PySpark

SQL

Delta Lake

Bronze/Silver/Gold

AWS

S3

IAM

Glue

Lambda

Kinesis

Data modelling

Data quality

Performance optimisation

CI/CD

Git

Databricks Workflows

Delta Live Tables

Unity Catalog

Auto Loader

Structured Streaming

APIs ingestion

XML/JSON ingestion

Tools

Git-based tools

Job description

PERSOL Outsourcing is seeking a skilled Data Engineer to design and operate batch and streaming pipelines using Databricks and AWS. You will work across Bronze/Silver/Gold layers, building scalable data processing with PySpark and Delta Lake.

Join a fast-paced team, implement robust data ingestion from APIs and files, ensure data quality, performance, and reliability, and contribute to CI/CD and production deployments.