Python Data Engineer – Banking, Spark & ETL

Unison Consulting Pte Ltd; Singapore; On-site; SGD 70,000 - 100,000; Full time

Job summary

A leading consulting firm in Singapore is seeking an experienced Python Developer specializing in Data Engineering. The role involves designing and supporting scalable data pipelines within a large enterprise banking environment. Responsibilities include developing ETL processes, implementing distributed data processing using Apache Spark, optimizing SQL and HiveQL, and ensuring data governance and security standards are met. The ideal candidate should have a strong background in data engineering, excellent collaboration skills, and a commitment to quality practices in data management.

Qualifications

• Strong experience with data pipelines and analytics platforms.
• Proven track record in end-to-end data engineering.
• Familiarity with data governance and security practices.

Responsibilities

• Develop robust data ingestion, transformation, and loading processes.
• Implement distributed data processing using Apache Spark.
• Design and maintain logical and physical data models.
• Write and optimize SQL and HiveQL queries.
• Schedule and monitor data pipelines ensuring SLA adherence.
• Perform performance tuning for scalability and efficiency.
• Implement data validation and lineage checks.
• Build operational dashboards and alerts.
• Maintain comprehensive documentation and coding standards.
• Apply data privacy and security best practices.
• Drive continuous service and process improvements.
• Prepare unit test cases and work closely with testing teams.
• Package and migrate code across environments with governance.

Skills

Data ingestion, transformation, and loading (ETL/ELT)

Apache Spark (PySpark / Scala)

SQL

HiveQL

Data validation and reconciliation

Performance tuning

Data privacy and security best practices

Tools

Control-M

Job description

A leading consulting firm in Singapore is seeking an experienced Python Developer specializing in Data Engineering. The role involves designing and supporting scalable data pipelines within a large enterprise banking environment. Responsibilities include developing ETL processes, implementing distributed data processing using Apache Spark, optimizing SQL and HiveQL, and ensuring data governance and security standards are met. The ideal candidate should have a strong background in data engineering, excellent collaboration skills, and a commitment to quality practices in data management.