Senior Data Engineer – PySpark, Databricks & Data Lakehouse

D L Resources Pte Ltd

Singapore

On-site

SGD 120,000 - 180,000

Full time

2 days ago

Be an early applicant

Application generator

Get a reply from this employer — a resume and cover letter tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Bank Sector Client in Singapore seeks a Senior Data Engineer / Lakehouse Engineer to design, build, and operate enterprise data platforms and data marketplaces. You will enable data products, data contracts, and governance across batch, streaming, and API-based ingestion.

Role emphasizes Spark, Python, SQL, and cloud data platforms with strong DevSecOps, CI/CD, and AI-first data pipelines capabilities. You will collaborate across engineering, analytics, and AI teams.

Qualifications

8–12 years in Data Engineering, Big Data, Lakehouse, or Data Warehouse.
Hands-on with enterprise data platforms such as Databricks, Snowflake, or cloud providers.
Experience building data products, pipelines, and data marketplaces.

Responsibilities

Design, build, and operationalize enterprise Data Lake/Lakehouse platforms and data products.
Develop scalable ingestion and processing pipelines for batch, streaming, CDC, and API-based integration.
Create multimodal ingestion including extraction from documents, images, video, and audio.
Implement data contracts, SLAs, data quality and governance frameworks.
Expose data via APIs, dashboards, and data products; support AI/ML data pipelines.
Collaborate across distributed teams and ensure DevSecOps and CI/CD practices.

Skills

Data engineering

Spark

Python

SQL

Databricks

Cloud platforms

ETL

CI/CD

Education

Bachelor's degree in CS/Engineering/IT/Data Science

Tools

Apache Iceberg

Apache Hudi

Delta Lake

Kafka

Airflow

Kubernetes

Docker

Terraform

Job description

Client: Bank Sector Client

About the Role

D L Resources is supporting a leading banking-sector client in hiring an experienced Senior Data Engineer / Lakehouse Engineer to design, build, and operationalize enterprise-scale data platforms, Lakehouse solutions, data products, and data marketplace capabilities.

The successful candidate will have strong hands‑on expertise in modern data engineering technologies, distributed processing, cloud data platforms, streaming, open table formats, and DevSecOps practices. The role will also contribute to emerging Generative AI, RAG, vector search, multimodal data processing, and AI-driven data pipelines.

Key Responsibilities

Design, implement, and operationalize enterprise Data Lake/Lakehouse platforms, data products, and data marketplace capabilities.
Build scalable data ingestion and processing pipelines supporting batch, streaming, CDC, event-driven, and API-based integration patterns.
Develop multimodal and unstructured data ingestion pipelines, including:
- Content extraction from multiple document and file formats.
- Metadata and field extraction using regular expressions and other processing techniques.
- Extraction and processing of embedded images.
- Video frame extraction and processing.
- Audio and transcript extraction.
Build, test, deploy, and maintain foundation and business data products based on agreed data contracts, SLAs, reconciliation rules, and data quality controls.
Implement and manage modern open table formats including Apache Iceberg, Apache Hudi, and Delta Lake.
Develop data pipelines and platform capabilities supporting RAG, vector search, Generative AI, NLP, and agentic AI use cases.
Optimize Spark and distributed data workloads for scalability, reliability, cost, and performance.
Perform production troubleshooting, performance tuning, root‑cause analysis, and operational support.
Implement data transformation, reconciliation, metadata, lineage, governance, and data quality frameworks.
Expose and distribute data through APIs, event streams, dashboards, BI platforms, and data products.
Develop internal engineering tools and supporting applications using Python, shell scripting, APIs, and modern web frameworks where required.
Create and maintain technical architecture documentation, deployment guides, operating procedures, and production runbooks.
Ensure solutions comply with enterprise engineering standards, security requirements, DevSecOps controls, CI/CD practices, and software delivery standards.
Collaborate with distributed engineering, architecture, analytics, AI/ML, business, and technology teams across multiple initiatives.

Required Experience & Technical Skills

8–12 years of experience in Data Engineering, Big Data, Data Lake, Data Warehouse, or Lakehouse implementations.
Strong hands‑on experience with one or more enterprise data platforms such as:
Databricks, Snowflake, Cloudera, Microsoft Azure, AWS, Google Cloud Platform (GCP), Huawei Cloud, or Alibaba Cloud.
Strong experience designing and developing enterprise data products and/or data marketplace solutions.
Advanced hands‑on expertise in:
Apache Spark, PySpark, SQL, Python, and/or Scala.
Strong programming skills in one or more of:
Python, Scala, Java, and SQL.
Experience implementing open table formats such as Apache Iceberg, Apache Hudi, and Delta Lake, together with cloud/object storage platforms.
Proven experience building enterprise frameworks for:
data ingestion, transformation, reconciliation, validation, and data quality.
Hands‑on experience with relevant distributed data and integration technologies such as:
Kafka, Flink, Spark Streaming, Airflow, Trino, Dremio, Hive, and Impala.
Strong experience with containerization, orchestration, infrastructure automation, and CI/CD technologies including:
Kubernetes, OpenShift, Docker, Terraform, Jenkins, Git, and CI/CD pipelines.
Experience implementing monitoring, logging, observability, and production support capabilities for enterprise data platforms.
Strong understanding of data modelling, metadata management, data lineage, governance, and data security.
Experience designing architectures for structured, semi‑structured, and unstructured data across Data Lake, Lakehouse, and Data Warehouse environments.
Experience exposing data through REST APIs, event streams, dashboards, BI platforms, and other consumption channels.

AI / ML & Advanced Data Engineering Experience

Experience in one or more of the following areas will be highly advantageous:

Building data architectures supporting NLP, Generative AI, RAG, vector databases/vector search, and AI-driven analytics.
Ingestion, extraction, curation, enrichment, and governance of unstructured and multimodal data.
Experience with ML platforms and frameworks such as MLflow, Cloudera Machine Learning (CML), Spark MLlib, scikit‑learn, and XGBoost.
Experience supporting machine‑learning model deployment and operationalization.
Building internal engineering applications or tools using Python, Flask, React, shell scripting, or similar technologies.

Additional Advantageous Experience

Experience with enterprise migration or modernization involving Teradata, Netezza, Greenplum, or other MPP data warehouse technologies.
Experience working within large‑scale banking, financial services, regulated, or enterprise environments.
Understanding of enterprise data governance, security, privacy, and regulatory requirements.

Education

Bachelor’s degree in Computer Science, Engineering, Information Technology, Data Science, or a related discipline.

Preferred Certifications

Relevant professional certifications are advantageous, including:

Databricks Certified Data Engineer
Microsoft Azure Data Engineer certification
AWS Data Engineering / Analytics certification
Google Cloud Professional Data Engineer
Snowflake SnowPro Certification
DAMA Certified Data Management Professional (CDMP)

What Will Help You Succeed

Strong engineering, automation, and problem‑solving mindset.
Excellent troubleshooting, performance‑tuning, and root‑cause‑analysis capabilities.
Ability to design solutions for high‑volume, highly available, enterprise‑scale data environments.
Strong communication and stakeholder‑management skills.
Ability to collaborate effectively across distributed teams and manage multiple concurrent initiatives.
Experience working within Agile and DevSecOps delivery environments.
Strong commitment to engineering quality, security, operational excellence, and continuous improvement.