Big Data Engineer (Hadoop)

Nicheton Consulting LLP; Bengaluru; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

CoHire is hiring a Big Data / Platform Engineer to support and extend our Hadoop-based Big Data platform, bring AI-enabled tooling into platform operations, and own platform reliability and automation.

You will collaborate with data engineers and architects, work on Spark/ETL workloads, scripting in Python or Bash, and help scale the platform as data volumes grow.

Qualifications

• Degree in Computer Science, Information Systems, or related field (or equivalent industry experience).
• Experience with Big Data ecosystems - Hadoop, Spark, ETL pipelines.
• Scripting/automation experience (e.g. Python, Bash, or similar) for operational tooling.
• Working knowledge of Linux systems administration.
• Experience with monitoring and alerting tools in a production data platform environment.
• Familiarity with applying AI/automation tools in a platform or data operations context.
• Understanding of opensource technology evaluation and adoption in an enterprise setting.

Responsibilities

• Support and extend the Hadoop-based Big Data platform, including Spark and ETL workloads.
• Add new platform engineering capability into the Big Data environment as requirements evolve.
• Test and troubleshoot the platform on an ongoing basis to maintain reliability.
• Monitor platform health, resource utilisation, and job performance, escalating and resolving issues as they arise.
• Support capacity planning for the platform as data volumes and workloads grow.
• Contribute to platform documentation, including configuration standards and operational runbooks.
• Work closely with data engineers, architects, and business stakeholders to understand platform requirements.
• Contribute technical input into design discussions for new platform capability.
• Support other engineers on the team with platform-related questions and troubleshooting.

Skills

Python

Bash

Linux administration

Agile delivery

Monitoring tools

AI tooling

Education

Bachelor's degree in Computer Science, Information Systems, or related field

Tools

Hadoop

Spark

ETL pipelines

AI tooling platforms

Monitoring & alerting tools

Linux

Job description

We are hiring a Big Data / Platform Engineer to support and extend the clients Hadoop-based Big Data platform, and to bring AI-enabled tooling into platform operations. The role covers platform build-out, reliability, and automation, working closely with engineering and data teams. This role is focused on platform engineering and automation Big Data & Platform Engineering

• Support and extend the Hadoop-based Big Data platform, including Spark and ETL workloads.
• Add new platform engineering capability into the Big Data environment as requirements evolve.
• Test and troubleshoot the platform on an ongoing basis to maintain reliability.
• Monitor platform health, resource utilisation, and job performance, escalating and resolving issues as they arise.
• Support capacity planning for the platform as data volumes and workloads grow.
• Contribute to platform documentation, including configuration standards and operational runbooks.

AI Tooling & Automation

• Implement AI technologies to support and automate platform operations.
• Evaluate and apply opensource technologies where they improve efficiency or reduce cost.
• Automate recurring operational tasks using AI-enabled tooling.
• Research and benchmark new tools and techniques against the existing platform to identify improvement opportunities.

Collaboration & Delivery

• Work closely with data engineers, architects, and business stakeholders to understand platform requirements.
• Contribute technical input into design discussions for new platform capability.
• Support other engineers on the team with platform-related questions and troubleshooting.
• Experience with Big Data ecosystems - Hadoop, Spark, ETL pipelines.
• Scripting/automation experience (e.g. Python, Bash, or similar) for operational tooling.
• Working knowledge of Linux systems administration.
• Experience with monitoring and alerting tools in a production data platform environment.
• Familiarity with applying AI/automation tools in a platform or data operations context.
• Understanding of opensource technology evaluation and adoption in an enterprise setting.

Functional Skills

• Experience working within Agile delivery methodology.
• Able to research and benchmark technology against best-in-class alternatives.
• Comfortable operating as a platform engineering profile focused on Big Data infrastructure.
• Banking, financial services, or other regulated enterprise environment experience preferred.
• Self-motivated, able to own and drive platform improvements with minimal supervision.

Experience & Education

• Degree in Computer Science, Information Systems, or related field (or equivalent industry experience).

Soft Skills

• Strong communication skills — able to clearly present ideas and technical approaches to the team.
• Comfortable questioning and querying ideas constructively, and engaging in open discussion with colleagues.
• Collaborative team player who works well across the wider technology team.