Databricks Data Engineer: Ingest & Spark Pipeline Expert

Percept Solutions Pte Ltd; Singapore; On-site; SGD 90,000 - 120,000; Full time

Job summary

Percept Solutions Pte Ltd is seeking a Data Engineer to design, build, and support reliable data ingestion and transformation pipelines on AWS and Databricks. You will collaborate with architects, engineers, QA, and business teams to deliver curated datasets and production-ready processes.

Ideal candidates have 3+ years in data engineering, hands-on experience with Databricks, Spark, PySpark, and Delta Lake, and a strong focus on data quality and performance.

Qualifications

• 3+ years of experience in data engineering or enterprise data-platform delivery.
• Hands-on experience with Databricks, Apache Spark, PySpark, and SQL.
• Experience with Delta Lake and layered data architecture (Bronze/Silver/Gold).
• Experience building production data pipelines with robust logging, error handling and recovery.

Responsibilities

• Design and build batch and/or streaming data pipelines using Databricks and AWS services.
• Develop data processing across Bronze, Silver, and Gold layers using SQL, Python/PySpark and Delta Lake.
• Implement ingestion from APIs, files, databases, object storage, or other data sources.
• Develop data transformations and reusable data-engineering components.
• Implement data validation, logging, error handling, rerun/recovery, and monitoring.
• Optimize Spark jobs, clusters, partitions, and storage for performance and cost.
• Support CI/CD, environment promotion, SIT/UAT, production deployment, and documentation.
• Collaborate with platform, application, BI, and data-governance teams.

Skills

Databricks

Apache Spark

PySpark

SQL

Data modelling

Data quality

Performance optimisation

Tools

Databricks

Delta Lake

AWS S3

AWS Glue

AWS Lambda

AWS Kinesis

Job description

Percept Solutions Pte Ltd is seeking a Data Engineer to design, build, and support reliable data ingestion and transformation pipelines on AWS and Databricks. You will collaborate with architects, engineers, QA, and business teams to deliver curated datasets and production-ready processes.

Ideal candidates have 3+ years in data engineering, hands-on experience with Databricks, Spark, PySpark, and Delta Lake, and a strong focus on data quality and performance.