Databricks Engineer – 12 months contract

PERSOL TECH SERVICES PTE. LTD.; Singapore; On-site; SGD 90,000 - 150,000; Full time

Job summary

PERSOL TECH SERVICES PTE. LTD. is seeking a data engineer to design and build batch and streaming data pipelines using Databricks and AWS services.

You will develop data processing across Bronze, Silver, and Gold layers with SQL, Python/PySpark, and Delta Lake, and implement ingestion from APIs, files, and databases. The role requires building reusable data transformations, robust data validation, and scalable logging/monitoring.

Qualifications

• 3+ years of experience in data engineering or enterprise data-platform delivery.
• Hands-on experience with Databricks, Apache Spark, PySpark, and SQL.
• Experience with Delta Lake and Bronze/Silver/Gold data layers.
• Experience building production data pipelines with logging, error handling, and recovery.
• Experience with AWS services such as S3, IAM, Glue, Lambda, and Kinesis.
• Strong understanding of data modelling, data quality, and performance optimisation.
• Knowledge of Databricks Workflows, Delta Live Tables, Unity Catalog, Auto Loader, or Structured Streaming.
• Experience with APIs, XML/JSON, event or near-real-time ingestion.
• CI/CD and infrastructure/configuration automation using Git-based tools.
• Experience with Tableau, BI datasets, or data warehouse integration.
• Certifications related to Databricks, AWS, data engineering, AI/ML, or GenAI are a plus.
• Strong engineering fundamentals, problem-solving, and attention to data correctness.

Responsibilities

• Design and build batch and/or streaming data pipelines using Databricks and AWS services.
• Develop data processing across Bronze, Silver, and Gold layers using SQL, Python/PySpark, and Delta Lake.
• Implement ingestion from APIs, files, databases, object storage, or other enterprise data sources.
• Develop data transformations, dimensional or curated datasets, and reusable data-engineering components.
• Implement data validation, reconciliation, error handling, logging, rerun/recovery, and monitoring.
• Optimise Spark jobs, clusters, partitioning, and storage for performance and cost.
• Support CI/CD, environment promotion, SIT/UAT, production deployment, documentation, and operational handover.
• Collaborate with platform, application, BI, and data-governance teams.

Job description

Key Responsibilities

• Design and build batch and/or streaming data pipelines using Databricks and AWS services.
• Develop data processing across Bronze, Silver, and Gold layers using SQL, Python/PySpark, and Delta Lake.
• Implement ingestion from APIs, files, databases, object storage, or other enterprise data sources.
• Develop data transformations, dimensional or curated datasets, and reusable data-engineering components.
• Implement data validation, reconciliation, error handling, logging, rerun/recovery, and monitoring.
• Optimise Spark jobs, clusters, partitioning, and storage for performance and cost.
• Support CI/CD, environment promotion, SIT/UAT, production deployment, documentation, and operational handover.
• Collaborate with platform, application, BI, and data-governance teams.

Requirements

• 3+ years of experience in data engineering or enterprise data-platform delivery.
• Hands-on experience with Databricks, Apache Spark, PySpark, and SQL.
• Experience with Delta Lake and layered data architecture such as Bronze/Silver/Gold.
• Experience building production data pipelines with proper logging, error handling, and recovery.
• Experience with AWS services such as S3, IAM, Glue, Lambda, Kinesis, or related services.
• Good understanding of data modelling, data quality, and performance optimisation.
• Databricks Workflows, Delta Live Tables, Unity Catalog, Auto Loader, or Structured Streaming.
• Experience with APIs, XML/JSON, event or near-real-time ingestion.
• CI/CD and infrastructure/configuration automation using Git-based tools.
• Experience with Tableau, BI datasets, or data warehouse integration.
• Databricks, AWS, data engineering, AI/ML, or GenAI-related certification(s).
• Strong engineering fundamentals and willingness to learn.
• Practical problem-solving and attention to data correctness.
• Clean implementation, documentation discipline, and reliable delivery.
• Ability to work in a fast-paced project environment and collaborate with different technical teams.
• Ownership of assigned tasks and openness to technical guidance and feedback.