Senior Data Engineer

Epergne Solutions; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Epergne Solutions is seeking a Senior Data Engineer in Singapore to design, develop, and maintain scalable data pipelines and platforms. You will work with AWS services, Databricks, PySpark, and Delta Lake to build reliable data lakehouse solutions.

Responsibilities include data ingestion, transformation, governance, and performance optimization while mentoring teammates and upholding engineering standards. A strong background in Spark, Cloud, and data modelling is essential.

Qualifications

• Strong practical experience with Python, PySpark and SQL.
• Hands-on with Databricks, Apache Spark and Delta Lake.
• Experience using AWS data services (S3, Glue, IAM, Lambda, Redshift).
• Knowledge of data lake/lakehouse architecture and data modelling.
• Understanding of CDC, data quality, workflow orchestration, and monitoring.

Responsibilities

• Design and develop batch and streaming data pipelines using AWS and Databricks.
• Build and maintain data lakehouse solutions with Delta Lake.
• Develop data ingestion, transformation and data modelling processes.
• Implement data quality checks, validation and governance.
• Automate workflows using Databricks and AWS services.
• Monitor and troubleshoot data pipelines and production issues.
• Optimise data processing performance and cloud costs.
• Implement data security, access controls and encryption.
• Review code, perform engineering standards checks and mentor team members.
• Maintain technical documentation and operational procedures.

Skills

Python

PySpark

SQL

Data pipelines

AWS

Education

Bachelor's degree in Computer Science or related field

Tools

Databricks

Apache Spark

Delta Lake

AWS (S3, Glue, IAM, Lambda, Redshift)

Git, CI/CD

Terraform or IaC

Job description

We are looking for a Senior Data Engineer with strong experience in AWS, Databricks, Python, PySpark and SQL to design, develop and maintain scalable data pipelines and data platforms.

Key Responsibilities

• Design and develop batch and streaming data pipelines using AWS and Databricks.
• Build and maintain data lakehouse solutions using Delta Lake.
• Develop data ingestion, transformation and data modelling processes.
• Implement data quality checks, validation and data governance.
• Automate workflows using Databricks and AWS services.
• Monitor and troubleshoot data pipelines and production issues.
• Optimise data processing performance and cloud costs.
• Implement data security, access controls and encryption.
• Review code, follow engineering standards and mentor team members.
• Maintain technical documentation and operational procedures.

Mandatory Skills

• Strong experience in Python, PySpark and SQL.
• Hands-on experience with Databricks, Apache Spark and Delta Lake.
• AWS data services such as S3, Glue, IAM, Lambda and Redshift.
• Experience in data lake/lakehouse architecture and data modelling.
• Knowledge of incremental data processing and Change Data Capture (CDC).
• Experience with data quality, workflow orchestration and monitoring.
• Knowledge of Git, CI/CD and Terraform or equivalent IaC tools.
• Experience in performance tuning, troubleshooting and cloud security.
• Technical leadership, code reviews or mentoring experience.

Preferred Skills

• Databricks Unity Catalog, Lakeflow and Structured Streaming.
• Apache Airflow, dbt, Kafka or AWS Kinesis.
• Experience with ML/AI data pipelines.
• Relevant AWS or Databricks certifications.

Education

Bachelor?s degree in Computer Science, Information Technology, Data Engineering or a related field.