Databricks Engineer – 12 months contract

PERSOL Outsourcing; Singapore; On-site; SGD 90,000 - 150,000; Full time

Job summary

PERSOL Outsourcing is seeking a skilled Data Engineer to design and operate batch and streaming pipelines using Databricks and AWS. You will work across Bronze/Silver/Gold layers, building scalable data processing with PySpark and Delta Lake.

Join a fast-paced team, implement robust data ingestion from APIs and files, ensure data quality, performance, and reliability, and contribute to CI/CD and production deployments.

Qualifications

• 3+ years in data engineering or enterprise data-platform delivery.
• Hands-on with Databricks, Apache Spark, PySpark, and SQL.
• Experience with Delta Lake and Bronze/Silver/Gold data architecture.
• Production data pipelines with logging, error handling, and recovery.
• Experience with AWS services such as S3, IAM, Glue, Lambda, Kinesis.
• Strong data modelling, data quality, and performance optimisation.
• CI/CD and infrastructure/configuration automation using Git-based tools.
• Experience with APIs, XML/JSON ingestion.

Responsibilities

• Design and build batch and streaming data pipelines using Databricks and AWS.
• Develop data processing across Bronze, Silver, and Gold layers using SQL, Python/PySpark, and Delta Lake.
• Implement ingestion from APIs, files, databases, object storage, or other enterprise data sources.
• Develop data transformations, dimensional or curated datasets, and reusable data-engineering components.
• Implement data validation, reconciliation, error handling, logging, rerun/recovery, and monitoring.
• Optimise Spark jobs, clusters, partitioning, and storage for performance and cost.
• Support CI/CD, environment promotion, SIT/UAT, production deployment, documentation, and operational handover.
• Collaborate with platform, application, BI, and data-governance teams.

Skills

Databricks

Apache Spark

PySpark

SQL

Delta Lake

Bronze/Silver/Gold

AWS

S3

IAM

Glue

Lambda

Kinesis

Data modelling

Data quality

Performance optimisation

CI/CD

Git

Databricks Workflows

Delta Live Tables

Unity Catalog

Auto Loader

Structured Streaming

APIs ingestion

XML/JSON ingestion

Tools

Git-based tools

Job description

Key Responsibilities

• Design and build batch and/or streaming data pipelines using Databricks and AWS services.
• Develop data processing across Bronze, Silver, and Gold layers using SQL, Python/PySpark, and Delta Lake.
• Implement ingestion from APIs, files, databases, object storage, or other enterprise data sources.
• Develop data transformations, dimensional or curated datasets, and reusable data-engineering components.
• Implement data validation, reconciliation, error handling, logging, rerun/recovery, and monitoring.
• Optimise Spark jobs, clusters, partitioning, and storage for performance and cost.
• Support CI/CD, environment promotion, SIT/UAT, production deployment, documentation, and operational handover.
• Collaborate with platform, application, BI, and data-governance teams.

Requirements

• 3+ years of experience in data engineering or enterprise data-platform delivery.
• Hands-on experience with Databricks, Apache Spark, PySpark, and SQL.
• Experience with Delta Lake and layered data architecture such as Bronze/Silver/Gold.
• Experience building production data pipelines with proper logging, error handling, and recovery.
• Experience with AWS services such as S3, IAM, Glue, Lambda, Kinesis, or related services.
• Good understanding of data modelling, data quality, and performance optimisation.
• Databricks Workflows, Delta Live Tables, Unity Catalog, Auto Loader, or Structured Streaming.
• Experience with APIs, XML/JSON, event or near-real-time ingestion.
• CI/CD and infrastructure/configuration automation using Git-based tools.
• Experience with Tableau, BI datasets, or data warehouse integration.
• Databricks, AWS, data engineering, AI/ML, or GenAI-related certification(s).
• Strong engineering fundamentals and willingness to learn.
• Practical problem-solving and attention to data correctness.
• Clean implementation, documentation discipline, and reliable delivery.
• Ability to work in a fast-paced project environment and collaborate with different technical teams.
• Ownership of assigned tasks and openness to technical guidance and feedback.