AI-Driven Data Engineer: Databricks + AWS Pipelines

PERSOL; Singapore; Vor Ort; SGD 90.000 - 130.000; Vollzeit

Zusammenfassung

PERSOL is seeking a Data Engineer in Singapore to design, build, and support data ingestion and transformation pipelines on AWS and Databricks. You will collaborate with architects, engineers, QA, and business teams to deliver reliable data pipelines, curated datasets, and production-ready operations.

The role requires hands-on experience with Databricks, Apache Spark, PySpark, SQL, and Delta Lake, plus knowledge of data modelling and data quality controls.

Qualifikationen

• 3+ years of experience in data engineering or enterprise data-platform delivery.
• Hands-on experience with Databricks, Apache Spark, PySpark, and SQL.
• Experience with Delta Lake and Bronze/Silver/Gold layered data architecture.
• Experience building production data pipelines with logging, error handling and recovery.
• Experience with AWS services such as S3, IAM, Glue, Lambda, Kinesis.
• Strong understanding of data modelling, data quality, and performance optimisation.
• Experience with Databricks Workflows, Delta Live Tables, Unity Catalog, Auto Loader, or Structured Streaming.
• Experience with APIs, XML/JSON, event or near-real-time ingestion.
• Experience with CI/CD and infrastructure/configuration automation using Git-based tools.
• Experience with Tableau, BI datasets, or data warehouse integration.
• Databricks, AWS, Data Engineering, AI/ML, or GenAI certifications

Aufgaben

• Design and build batch and/or streaming data pipelines using Databricks and AWS services.
• Develop data processing across Bronze, Silver, and Gold layers with SQL, Python/PySpark, and Delta Lake.
• Implement ingestion from APIs, files, databases, object storage, or other sources.
• Develop data transformations and reusable data-engineering components.
• Implement data validation, logging, error handling, and recovery workflows.
• Optimise Spark jobs, clusters, partitioning, and storage for performance and cost.
• Support CI/CD, SIT/UAT, production deployment, and documentation.
• Collaborate with platform, application, BI, and data-governance teams.

Kenntnisse

Databricks

Apache Spark

PySpark

SQL

AWS services

Delta Lake

Bronze/Silver/Gold

Data modelling

Data quality

CI/CD

APIs/XML/JSON

Structured Streaming

Automation/Git

Tableau

BI datasets

Data warehouse integration

Databricks Workflows

Delta Live Tables

Unity Catalog

Tools

Delta Lake

Databricks Workflows

Unity Catalog

Auto Loader

Structured Streaming

Kinesis

Jobbeschreibung

PERSOL is seeking a Data Engineer in Singapore to design, build, and support data ingestion and transformation pipelines on AWS and Databricks. You will collaborate with architects, engineers, QA, and business teams to deliver reliable data pipelines, curated datasets, and production-ready operations.

The role requires hands-on experience with Databricks, Apache Spark, PySpark, SQL, and Delta Lake, plus knowledge of data modelling and data quality controls.