Senior Data Engineer: Build AWS/Databricks Pipelines

Epergne Solutions; Singapore; In loco; SGD 120.000 - 180.000; Tempo pieno

Descrizione del lavoro

Epergne Solutions is seeking a Senior Data Engineer in Singapore to design, develop, and maintain scalable data pipelines and platforms. You will work with AWS services, Databricks, PySpark, and Delta Lake to build reliable data lakehouse solutions.

Responsibilities include data ingestion, transformation, governance, and performance optimization while mentoring teammates and upholding engineering standards. A strong background in Spark, Cloud, and data modelling is essential.

Competenze

• Strong practical experience with Python, PySpark and SQL.
• Hands-on with Databricks, Apache Spark and Delta Lake.
• Experience using AWS data services (S3, Glue, IAM, Lambda, Redshift).
• Knowledge of data lake/lakehouse architecture and data modelling.
• Understanding of CDC, data quality, workflow orchestration, and monitoring.

Mansioni

• Design and develop batch and streaming data pipelines using AWS and Databricks.
• Build and maintain data lakehouse solutions with Delta Lake.
• Develop data ingestion, transformation and data modelling processes.
• Implement data quality checks, validation and governance.
• Automate workflows using Databricks and AWS services.
• Monitor and troubleshoot data pipelines and production issues.
• Optimise data processing performance and cloud costs.
• Implement data security, access controls and encryption.
• Review code, perform engineering standards checks and mentor team members.
• Maintain technical documentation and operational procedures.

Conoscenze

Python

PySpark

SQL

Data pipelines

AWS

Formazione

Bachelor's degree in Computer Science or related field

Strumenti

Databricks

Apache Spark

Delta Lake

AWS (S3, Glue, IAM, Lambda, Redshift)

Git, CI/CD

Terraform or IaC

Descrizione del lavoro

Epergne Solutions is seeking a Senior Data Engineer in Singapore to design, develop, and maintain scalable data pipelines and platforms. You will work with AWS services, Databricks, PySpark, and Delta Lake to build reliable data lakehouse solutions.

Responsibilities include data ingestion, transformation, governance, and performance optimization while mentoring teammates and upholding engineering standards. A strong background in Spark, Cloud, and data modelling is essential.