Data Engineer, AI Integration Specialist

PERSOL; Singapore; On-site; SGD 90,000 - 130,000; Full time

Job summary

PERSOL is seeking a Data Engineer in Singapore to design, build, and support data ingestion and transformation pipelines on AWS and Databricks. You will collaborate with architects, engineers, QA, and business teams to deliver reliable data pipelines, curated datasets, and production-ready operations.

The role requires hands-on experience with Databricks, Apache Spark, PySpark, SQL, and Delta Lake, plus knowledge of data modelling and data quality controls.

Qualifications

• 3+ years of experience in data engineering or enterprise data-platform delivery.
• Hands-on experience with Databricks, Apache Spark, PySpark, and SQL.
• Experience with Delta Lake and Bronze/Silver/Gold layered data architecture.
• Experience building production data pipelines with logging, error handling and recovery.
• Experience with AWS services such as S3, IAM, Glue, Lambda, Kinesis.
• Strong understanding of data modelling, data quality, and performance optimisation.
• Experience with Databricks Workflows, Delta Live Tables, Unity Catalog, Auto Loader, or Structured Streaming.
• Experience with APIs, XML/JSON, event or near-real-time ingestion.
• Experience with CI/CD and infrastructure/configuration automation using Git-based tools.
• Experience with Tableau, BI datasets, or data warehouse integration.
• Databricks, AWS, Data Engineering, AI/ML, or GenAI certifications

Responsibilities

• Design and build batch and/or streaming data pipelines using Databricks and AWS services.
• Develop data processing across Bronze, Silver, and Gold layers with SQL, Python/PySpark, and Delta Lake.
• Implement ingestion from APIs, files, databases, object storage, or other sources.
• Develop data transformations and reusable data-engineering components.
• Implement data validation, logging, error handling, and recovery workflows.
• Optimise Spark jobs, clusters, partitioning, and storage for performance and cost.
• Support CI/CD, SIT/UAT, production deployment, and documentation.
• Collaborate with platform, application, BI, and data-governance teams.

Skills

Databricks

Apache Spark

PySpark

SQL

AWS services

Delta Lake

Bronze/Silver/Gold

Data modelling

Data quality

CI/CD

APIs/XML/JSON

Structured Streaming

Automation/Git

Tableau

BI datasets

Data warehouse integration

Databricks Workflows

Delta Live Tables

Unity Catalog

Tools

Delta Lake

Databricks Workflows

Unity Catalog

Auto Loader

Structured Streaming

Kinesis

Job description

About the role

We are looking for a Data Engineer to design, build, and support reliable data ingestion and transformation pipelines on AWS and Databricks. The role will work with architects, engineers, QA, and business teams to deliver data pipelines, curated datasets, data-quality controls, and production-ready operational processes.

Key responsibilities

• Design and build batch and/or streaming data pipelines using Databricks and AWS services.
• Develop data processing across Bronze, Silver, and Gold layers using SQL, Python/PySpark, and Delta Lake.
• Implement ingestion from APIs, files, databases, object storage, or other enterprise data sources.
• Develop data transformations, dimensional or curated datasets, and reusable data-engineering components.
• Implement data validation, reconciliation, error handling, logging, rerun/recovery, and monitoring.
• Optimise Spark jobs, clusters, partitioning, and storage for performance and cost.
• Support CI/CD, environment promotion, SIT/UAT, production deployment, documentation, and operational handover.
• Collaborate with platform, application, BI, and data-governance teams.

About you

• 3+ years of experience in data engineering or enterprise data-platform delivery.
• Hands-on experience with Databricks, Apache Spark, PySpark, and SQL.
• Experience with Delta Lake and layered data architecture such as Bronze/Silver/Gold.
• Experience building production data pipelines with proper logging, error handling, and recovery.
• Experience with AWS services such as S3, IAM, Glue, Lambda, Kinesis, or related services.
• Good understanding of data modelling, data quality, and performance optimisation.
• Experience with Databricks Workflows, Delta Live Tables, Unity Catalog, Auto Loader, or Structured Streaming.
• Experience with APIs, XML/JSON, event or near-real-time ingestion.
• Experience with CI/CD and infrastructure/configuration automation using Git-based tools.
• Experience with Tableau, BI datasets, or data warehouse integration.
• Databricks, AWS, Data Engineering, AI/ML, or GenAI certifications.