Solution Architect

RISKDATA CONSULTING PTE. LTD.; Singapore; In loco; SGD 120.000 - 180.000; Tempo pieno

Descrizione del lavoro

Riskdata Consulting Pte. Ltd. seeks a senior Data Engineer to lead enterprise data platform initiatives in the banking sector, leveraging Hadoop/Cloudera expertise.

You will design and optimize scalable data pipelines, ensure data quality, and collaborate with cross-functional teams to deliver complex projects on time. This role demands 10+ years of hands-on delivery and strong stakeholder management.

Competenze

• Extensive experience with enterprise data platforms and data lake architectures.
• Proven track record delivering large-scale data delivery initiatives in banking/financial services.
• Strong hands-on with Hadoop ecosystem and data pipelines.
• Experience in CI/CD and DevOps practices.
• Excellent stakeholder management and project delivery skills.
• Experience in data governance and quality management.

Mansioni

• Design, develop, and maintain scalable data pipelines within the Enterprise Data Lake platform.
• Develop high-performance batch and near real-time ingestion frameworks.
• Design and implement transformation logic using Spark, HiveQL, SQL, and Java.
• Build data ingestion patterns for structured, semi-structured, and unstructured data.
• Maintain batch and near real-time data integration solutions.
• Monitor pipeline performance and production incidents; support deployments.
• Provide L3 production support and incident root cause analysis.
• Collaborate with platform teams for cluster utilization and workload management.

Conoscenze

Cloudera CDP

Hadoop

Spark

Hive

Java

Python

SQL

Kafka

Unix/Linux scripting

Shell scripting

ETL/ELT

Data modelling

Strumenti

Git

Jenkins

Azure DevOps

CI/CD pipelines

Descrizione del lavoro

This role requires strong technical knowledge of enterprise data platforms, excellent stakeholder management skills, and proven experience delivering complex projects within the banking and financial services industry.

10+ years of experience.

5+ years managing Enterprise Data Lake, Data Warehouse or Big Data delivery initiatives.

Proven experience in Hadoop/Cloudera environments.

Experience managing multiple concurrent project deliveries and BAU portfolios.

Banking and Financial Services experience preferred.

Data Engineering & Integration

• Design, develop, and maintain scalable data pipelines within the Enterprise Data Lake platform using Cloudera CDP, Hadoop, Spark, Hive, Java, Python, and SQL.
• Develop high-performance batch and near real-time data ingestion frameworks for onboarding enterprise source systems into the Data Lake.
• Develop complex transformation logic using Spark, HiveQL, SQL, and Java-based processing frameworks.
• Design and implement data ingestion patterns supporting structured, semi-structured, and unstructured datasets.
• Build and maintain batch and near real-time data integration solutions.
• Implement source-to-target mappings and ingestion specifications.

Hadoop & Cloudera Platform Development

• Develop and support solutions within the Cloudera Data Platform ecosystem.
• Build and maintain data pipelines leveraging:
• Hadoop HDFS
• Hive
• Impala
• Spark
• Kafka
• YARN, Oozie
• Perform deployment, troubleshooting, and performance tuning of applications running on the Hadoop ecosystem.
• Partner with platform teams to ensure optimal cluster utilization and workload management.

Data Quality & Governance

• Ensure adherence to Data Lake onboarding standards and governance requirements.
• Investigate and resolve data quality issues and production incidents.
• Ensure compliance with enterprise data governance standards.

DevOps & Platform Automation

• Integrate data engineering solutions into CI/CD pipelines using Git, Jenkins, Azure DevOps, or equivalent tools.
• Automate deployment, testing, monitoring, and operational support activities.
• Develop reusable frameworks and utilities to improve engineering productivity and platform scalability.

Production Support & Incident Management

• Provide L3 support for Data Lake application.
• Perform root cause analysis for production incidents and implement permanent fixes.
• Monitor pipeline execution, cluster health, and workload performance.
• Participate in release implementation, deployment validation, and post-production support activities.
• Strong hands on experience with Hadoop ecosystem and data pipeline orchestration.
• Hadoop/HDFS
• Hive
• Spark
• Unix/Linux scripting
• Shell scripting
• ETL/ELT development experience.
• Data modelling fundamentals.
• Performance tuning and optimization.
• Experience in Data Warehousing and Data Lake environments.
• Strong Java / Python competency with experiences.
• Strong SQL development skills.