Data Engineer / DevOps Engineer

ECLATPRIME DIGITAL PTE. LTD.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

ECLATPRIME DIGITAL PTE. LTD. in Singapore seeks a senior Data Engineer to design and build scalable data pipelines with Lakeflow and Delta Lake, leveraging Bronze/Silver/Gold medallions.

You will optimize performance, enforce governance using Unity Catalog and ACLs, review code, and implement real-time or near-real-time ingestion. Strong Python and Spark experience plus Databricks certification are highly valued.

Qualifications

• Total Data Engineering: 5+ years
• Databricks Certified Data Engineer Professional.
• Highly Preferred: Apache Spark Developer Associate
• Cloud (Optional): AWS certification.

Responsibilities

• Design & Build: Develop robust Medallion Architecture (Bronze/Silver/Gold) pipelines using Lakeflow and Delta Lake
• Performance Tuning: Proactively monitor and tune clusters, jobs, and serverless compute to reduce latency and infrastructure costs
• Governance Integration: Ensure every data asset is correctly tagged, secured through ACL, and discoverable via Unity Catalog
• Code Review: Enforce software engineering best practices (unit testing, modular code, and documentation) within the data team
• Streaming & Ingestion: Implement real-time or near-real-time data ingestion using Structured Streaming and Lake flow Connect
• Observability: Monitor pipeline health and costs using system tables; set up alerts and dashboards for proactive issue detection.

Skills

Data Engineering

Databricks

Python

Spark

Hadoop

Streaming

Tools

Lakeflow

Delta Lake

Unity Catalog

Open-Source Apache Spark

Flink

Job description

Experience Requirements

• Total Data Engineering: 5+ years
• Databricks Specific: 1+ years of hands-on experience in production environments
• Enterprise Scale: Experience building enterprise-level scalable pipelines with Spark or Hadoop-based systems

Required Certifications

• Mandatory: Databricks Certified Data Engineer Professional.
• Highly Preferred: Apache Spark Developer Associate
• Cloud (Optional): Professional-level certification in AWS.

Key Responsibilities

• Design & Build: Develop robust Medallion Architecture (Bronze/Silver/Gold) pipelines using Lakeflow and Delta Lake
• Performance Tuning: Proactively monitor and tune clusters, jobs, and serverless compute to reduce latency and infrastructure costs
• Governance Integration: Ensure every data asset is correctly tagged, secured through ACL, and discoverable via Unity Catalog
• Code Review: Enforce software engineering best practices (unit testing, modular code, and documentation) within the data team
• Streaming & Ingestion: Implement real-time or near-real-time data ingestion using Structured Streaming and Lake flow Connect
• Observability: Monitor pipeline health and costs using system tables; set up alerts and dashboards for proactive issue detection.

Preferred Candidate Background

• Distributed Systems: Proven track record with Open-Source Apache Spark, Flink, or Hadoop.
• Python-Centric: Advanced Python skills are a prerequisite; the project requires building complex logic that exceeds standard SQL capabilities.
• Engineering Mindset: Experience in refactoring legacy Big Data workloads into clean, testable, and optimized PySpark applications.