Senior Data Engineer - Lakehouse & Cloud Pipelines

OPTIMUM SOLUTIONS (SINGAPORE) PTE LTD; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

OPTIMUM SOLUTIONS (SINGAPORE) PTE LTD seeks a Senior Data Engineer to design, build, deploy and maintain reliable data pipelines on a cloud-based platform using AWS and Databricks. Lead data ingestion, transformation, quality, lineage and observability for analytics, reporting and ML workloads.

You will architect Bronze/Silver/Gold layers, implement incremental ingestion and CDC, and ensure scalable, secure data products. Strong Python, PySpark, SQL and AWS/Databricks experience are essential.

Qualifications

• Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related technical discipline.
• 6+ years of relevant data or software engineering experience, including hands‑on design and operation of production data pipelines and at least 2 years with cloud platforms.
• Strong Python, PySpark and SQL capabilities for distributed processing, transformations, testing and debugging.

Responsibilities

• Data Pipeline Architecture and Delivery: design and implement maintainable batch and streaming pipelines across AWS and Databricks.
• Lakehouse and Data Product Engineering: design Bronze, Silver, Gold data layers and reusable data products with clear ownership.
• Source Integration: implement ingestion from databases, files, APIs and event streams with full-load, incremental and CDC patterns.
• Data Quality and Contracts: enforce schemas, data contracts, automated validation and anomaly checks.
• Orchestration and Deployment: build parameterised workflows with Databricks Workflows or Lakeflow; integrate CI/CD and IaC.
• Metadata, Lineage and Governance: implement metadata, lineage and access controls using Unity Catalog, AWS Glue Data Catalog or Lake Formation.

Skills

Python

PySpark

SQL

Data modeling

Data quality

Data governance

CI/CD

Terraform

Git

Distributed processing

Education

Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or related technical discipline

Tools

Databricks

Apache Spark

Delta Lake

AWS S3

AWS Glue

Lake Formation

IAM

KMS

Lambda

Step Functions

EventBridge

CloudWatch

Redshift

MSK/Kinesis

Terraform

CI/CD

Git

Job description

OPTIMUM SOLUTIONS (SINGAPORE) PTE LTD seeks a Senior Data Engineer to design, build, deploy and maintain reliable data pipelines on a cloud-based platform using AWS and Databricks. Lead data ingestion, transformation, quality, lineage and observability for analytics, reporting and ML workloads.

You will architect Bronze/Silver/Gold layers, implement incremental ingestion and CDC, and ensure scalable, secure data products. Strong Python, PySpark, SQL and AWS/Databricks experience are essential.