Data Platform Engineer - Hadoop/Spark & Data Pipelines

Riskdata Consulting; Singapore; In loco; SGD 120.000 - 180.000; Tempo pieno

Descrizione del lavoro

Riskdata Consulting is seeking a Data Engineer to design, develop, and deploy scalable transformation frameworks across Hadoop platforms, building robust data pipelines, ingestion frameworks, and automation capabilities.

You will optimize data processing, ensure reliable integration of enterprise sources, and support data warehouse and analytics needs across banking domains, with exposure to streaming data and large-scale platforms.

Competenze

• Design, develop, and maintain scalable data pipelines within the Enterprise Data Lake platform using Cloudera CDP, Hadoop, Spark, Hive, Java, Python, and SQL.
• Develop high-performance batch and near real-time data ingestion frameworks for onboarding enterprise source systems into the Data Lake.
• Develop complex transformation logic using Spark, HiveQL, SQL, and Java-based processing frameworks.
• Design and implement data ingestion patterns supporting structured, semi-structured, and unstructured datasets.
• Build and maintain batch and near real-time data integration solutions.
• Implement source-to-target mappings and ingestion specifications.

Mansioni

• Develop and support solutions within the Cloudera Data Platform ecosystem.
• Build and maintain data pipelines leveraging Hadoop HDFS, Hive, Impala, Spark, Kafka, YARN, Oozie.
• Perform deployment, troubleshooting, and performance tuning of applications running on the Hadoop ecosystem.
• Partner with platform teams to ensure optimal cluster utilization and workload management.
• Ensure adherence to Data Lake onboarding standards and governance requirements.
• Investigate and resolve data quality issues and production incidents.
• Integrate data engineering solutions into CI/CD pipelines using Git, Jenkins, Azure DevOps or equivalent tools.

Strumenti

Cloudera CDP

Hadoop

Hive

Impala

Spark

Kafka

YARN

Oozie

HDFS

HiveQL

SQL

Java

Python

Unix/Linux scripting

Shell scripting

ETL/ELT

Data Warehousing

Data Lake

Descrizione del lavoro

Riskdata Consulting is seeking a Data Engineer to design, develop, and deploy scalable transformation frameworks across Hadoop platforms, building robust data pipelines, ingestion frameworks, and automation capabilities.

You will optimize data processing, ensure reliable integration of enterprise sources, and support data warehouse and analytics needs across banking domains, with exposure to streaming data and large-scale platforms.