Senior Data Engineer: Lakehouse & AI Pipelines

optimum solutions (singapore) pte ltd; Singapore; On-site; SGD 150,000 - 260,000; Full time

Job summary

optimum solutions (singapore) pte ltd is seeking an experienced Senior Data Engineer with 8–12 years of hands‑on expertise in Data Engineering, Big Data, Data Lake and Lakehouse platforms. The role focuses on building enterprise Lakehouse platforms, Data Products, and data marketplaces using Databricks, Spark, Python and SQL.

You will design scalable batch and streaming pipelines, implement data contracts, SLAs, and data quality controls, and work with Iceberg/Hudi/Delta Lake formats.

Qualifications

• 8–12 years of experience in Data Engineering, Big Data, Data Lake, or Lakehouse implementations.
• Hands‑on experience with Databricks, Snowflake, Cloudera, and cloud data platforms such as Azure, AWS, or GCP.
• Strong experience building Data Products and Data Marketplace capabilities.
• Excellent programming skills in Python, Scala, Java, and SQL.
• Strong hands‑on expertise with Spark / PySpark.
• Experience with Iceberg, Hudi, Delta Lake, and object storage platforms.
• Strong experience with data ingestion, transformation, reconciliation, and data quality frameworks.
• Experience with Kafka, Flink, Spark Streaming, Airflow, Trino, Dremio, Hive, and Impala.
• Hands‑on experience with Kubernetes, OpenShift, Docker, Terraform, Jenkins, Git, and CI/CD pipelines.
• Experience with MLflow and observability platforms/tools.
• Ability to design data architectures for NLP, AI/ML, GenAI, and unstructured data.
• Experience with ML platforms and libraries such as CML, Spark MLlib, scikit‑learn, and XGBoost, including model deployment.
• Experience developing internal engineering tools and applications using Python, Shell scripting, Flask, React, or similar modern frameworks.
• Knowledge of data modeling, metadata management, data lineage, governance, APIs, event streams, dashboards, and BI platforms.
• Experience with Teradata, Netezza, Greenplum, or MPP migration programs is an advantage.

Responsibilities

• Implement and operationalize enterprise Lakehouse platforms, Data Products, and Data Marketplace capabilities.
• Design and develop scalable batch, streaming, CDC, and API-based data ingestion pipelines.
• Build multimodal data ingestion pipelines supporting diverse data types.
• Build, test, and maintain foundation and business data products with defined data contracts, SLAs, and data quality controls.
• Implement and work with open table formats such as Apache Iceberg, Apache Hudi, and Delta Lake.
• Develop and support data pipelines for RAG, vector search, Generative AI, and agentic AI use cases.
• Perform performance tuning, optimization, production support, troubleshooting, and root cause analysis.
• Develop technical documentation, deployment guides, operational runbooks, and support procedures.
• Ensure compliance with engineering standards, DevSecOps controls, CI/CD practices, and software delivery standards.
• Collaborate with distributed engineering, architecture, data science, and business teams across multiple projects.

Skills

Python

SQL

Spark / PySpark

Cloud platforms

Data modeling

Data governance

CI/CD

DevSecOps

Tools

Databricks

Snowflake

Cloudera

Azure

AWS

GCP

Kubernetes

OpenShift

Docker

Terraform

Jenkins

Airflow

Kafka

Flink

Trino

Dremio

Hive

Impala

MLflow

Spark Streaming

Job description

optimum solutions (singapore) pte ltd is seeking an experienced Senior Data Engineer with 8–12 years of hands‑on expertise in Data Engineering, Big Data, Data Lake and Lakehouse platforms. The role focuses on building enterprise Lakehouse platforms, Data Products, and data marketplaces using Databricks, Spark, Python and SQL.

You will design scalable batch and streaming pipelines, implement data contracts, SLAs, and data quality controls, and work with Iceberg/Hudi/Delta Lake formats.