Applications Developer - Hadoop Data Pipelines & Lake

REGTECH INSIGHT PTE. LTD.; Singapore; Presencial; SGD 120.000 - 180.000; Jornada completa

Descripción de la vacante

REGTECH INSIGHT PTE. LTD. is seeking a skilled Data Engineer to design and deliver scalable data pipelines across Hadoop platforms, enabling enterprise data lakes, analytics, and downstream consumption. You will handle structured, semi-structured, and streaming data within critical banking domains.

The role emphasizes batch and near real-time ingestion, data quality, governance, and automation via CI/CD, Git, Jenkins, and Azure DevOps, ensuring reliable integration of diverse data sources.

Formación

• Design, develop, and maintain scalable data pipelines for Enterprise Data Lake.
• Develop high-performance batch and near real-time data ingestion frameworks.
• Implement data ingestion patterns for structured, semi-structured, and unstructured data.
• Ensure governance and quality in data onboarding and production systems.

Responsabilidades

• Design and maintain scalable data pipelines using CDP, Hadoop, Spark, Hive, Java, Python, and SQL.
• Develop real-time data ingestion frameworks for onboarding enterprise sources.
• Build data transformations with Spark, HiveQL, SQL, and Java-based processing.
• Automate deployment, testing, and monitoring within CI/CD pipelines.
• Provide L3 production support, troubleshoot incidents, and optimize performance.

Conocimientos

Data pipelines

Hadoop

Spark

Hive

Java

Python

SQL

CI/CD

Git

Jenkins

Azure DevOps

Unix/Linux scripting

ETL/ELT

Data Warehousing

Data Lake

Performance tuning

Data governance

Incident management

Kafka

Herramientas

Cloudera CDP

Hadoop

Hive

Impala

Spark

Kafka

YARN

Oozie

Descripción del empleo

REGTECH INSIGHT PTE. LTD. is seeking a skilled Data Engineer to design and deliver scalable data pipelines across Hadoop platforms, enabling enterprise data lakes, analytics, and downstream consumption. You will handle structured, semi-structured, and streaming data within critical banking domains.

The role emphasizes batch and near real-time ingestion, data quality, governance, and automation via CI/CD, Git, Jenkins, and Azure DevOps, ensuring reliable integration of diverse data sources.