Big Data Engineer

ACCORD INNOVATIONS PTE. LTD.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

ACCORD Innovations seeks an experienced Big Data Administrator / Big Data Engineer (L3) to support and maintain enterprise Big Data infrastructure in a production environment in Singapore. You will ensure availability, reliability and performance of Hadoop-based platforms, with emphasis on HDFS, YARN, Kafka and related technologies.

You will troubleshoot complex issues, manage large clusters, perform capacity planning and work with architects, developers and service managers to deliver

Qualifications

• Experience administering enterprise Hadoop clusters.
• Strong troubleshooting and incident management skills.
• Ability to work with architects and project teams on roadmaps and improvements.

Responsibilities

• Build, configure and maintain enterprise Hadoop cluster infrastructure.
• Monitor cluster health, availability and performance; tune workloads.
• Provide L3 production support and incident management for Big Data platforms.
• Collaborate with architects, developers and service managers on technology roadmaps and improvements.

Skills

Linux Administration

Hadoop Administration

HDFS

YARN

MapReduce

Shell / Perl / Python scripting

PAM

Kerberos

Access control and security mechanisms

Kafka

Tools

Hortonworks Data Platform (HDP)

Job description

Job Summary

We are looking for an experienced Big Data Administrator / Big Data Engineer - L3 to support and maintain enterprise Big Data infrastructure in a production environment.

The successful candidate will be responsible for the availability, reliability, performance and stability of Big Data platforms , with a strong focus on Hadoop ecosystem administration, Linux infrastructure, HDFS, YARN, Kafka and related technologies.

This is an L3 technical role requiring strong troubleshooting capabilities, hands‑on experience with large Hadoop clusters, production incident management, performance tuning, capacity planning and infrastructure improvements.

The role will also involve working closely with architects, developers, project teams, service managers and other technical teams to support technology roadmaps, transformation projects and continuous service improvement.

Key Responsibilities

Big Data Platform Administration

• Build, configure and maintain enterprise Hadoop cluster infrastructure .
• Perform Hadoop installation, upgrades, patches and version migrations.
• Administer large-scale Hadoop clusters and associated ecosystem components.
• Manage HDFS, YARN and MapReduce environments.
• Perform cluster maintenance, deployment and node addition/removal activities.
• Monitor cluster health, availability, capacity and performance.
• Perform HDFS backup and restoration activities.
• Manage file systems, storage capacity and cluster resources.
• Review Hadoop logs and monitoring alerts to identify and resolve issues.
• Ensure Hadoop clusters are properly configured, secured and optimized.
• Perform capacity planning and recommend infrastructure improvements.
• Tune cluster and MapReduce workloads for optimal performance.
• Configure and monitor cluster jobs and workloads.

Security & Access Management

• Manage Hadoop users and access permissions.
• Support Linux user administration related to Hadoop environments.
• Configure and support Kerberos authentication .
• Support PAM and other access‑control mechanisms.
• Monitor Hadoop cluster connectivity and security.
• Apply security patches and infrastructure hardening practices.
• Ensure security and operational procedures are followed.

Kafka & Data Platform Operations

• Perform Kafka administration and operational support .
• Monitor Kafka infrastructure and troubleshoot operational issues.
• Support availability and reliability of data‑streaming services.
• Work with development and architecture teams to resolve platform issues.
• Support data‑quality and data‑availability requirements across technology teams.

Linux & Infrastructure Administration

• Perform Linux system administration activities supporting Big Data platforms.
• Manage system monitoring, storage capacity and performance.
• Support OS‑level patches and infrastructure changes.
• Coordinate rolling OS changes with cluster administration tools.
• Automate software and application installation/configuration where applicable.
• Troubleshoot system and infrastructure issues using operating‑system and application logs.
• Support network and system administration activities within large‑scale data‑centre environments.

Scripting & Automation

• Develop and maintain automation scripts using Shell, Python or Perl .
• Debug existing scripts and automation processes.
• Automate installation, configuration and operational activities.
• Identify opportunities to improve operational efficiency through automation.

Production Support & Incident Management

• Provide L3 technical support for Big Data production environments.
• Lead the technical investigation of complex and high‑severity incidents.
• Perform root‑cause analysis and drive issues through to resolution.
• Proactively identify potential risks and service‑impacting issues.
• Support incident, problem and change‑management activities.
• Maintain system availability and reliability in accordance with agreed SLAs.
• Review technology changes and assess potential operational risks.
• Provide technical guidance to L1/L2 teams and partner resources.

Project & Technology Transformation

• Support Big Data infrastructure projects and transformation initiatives.
• Work with architects, developers, project teams and service managers on technology roadmaps.
• Ensure new solutions are production‑ready before operational handover.
• Support technical validation and readiness activities for new projects.
• Evaluate new technologies and identify opportunities for service improvement.
• Recommend improvements to Big Data infrastructure, processes and operational practices.
• Maintain technical documentation, SOPs, knowledge articles and best practices.
• Share technical knowledge and coach other team members where required.

Required Technical Skills

Critical Skills

• Linux Administration
• Hadoop Administration
• HDFS
• YARN
• MapReduce
• Hadoop ecosystem components
• Hortonworks Data Platform (HDP)
• Shell / Perl / Python scripting
• PAM
• Kerberos
• Access control and security mechanisms

Essential Skills

• Kafk