Senior Technology Engineer – DevOps Data Hadoop

D4 Insight; Bengaluru; On-site; INR 4,000,000 - 6,000,000; Full time

Job summary

D4 Insight in Bengaluru, India seeks a hands-on Tier-2 Production Engineer with deep Cloudera Data Platform expertise to install, secure, and optimize enterprise Hadoop/CDP environments. You will manage cluster administration, automation, and containerization while collaborating with data engineering teams.

The role requires 8+ years of experience, strong Linux fundamentals, and proficiency with Kubernetes/OpenShift, Terraform, Groovy, Ansible, and CI/CD tools.

Qualifications

• 8+ years of relevant experience with Cloudera/CDP administration.
• Hands-on production support for CDP environments.
• Experience automating data platforms using Shell, Terraform, Groovy, and Ansible.
• Experience with containerization and orchestration: Kubernetes/OpenShift.
• Strong Linux OS and networking fundamentals; proficient in Git and Agile practices.

Responsibilities

• Provide Tier-2 production support for Cloudera Data Platform.
• Install, configure, and optimize CDP, keeping versions updated.
• Implement security and compliance checks for data platforms.
• Collaborate with data engineering to optimize Spark, Hive, and YARN performance.
• Develop CI/CD pipelines and automation using GitHub Actions, Jenkins, ArgoCD.
• Support containerization with Podman/Docker and orchestration with Kubernetes/OpenShift.
• Drive observability and platform reliability.

Skills

Cloudera/CDP administration

Linux fundamentals

Agile engineering practices

Spark job performance

Tools

Shell

Terraform

Groovy

Ansible

Kubernetes

OpenShift

Podman

Docker

Git

Job description

Job Overview

We are looking for hands-on Tier-2 Production Engineer with strong Cloudera Data Platform (CDP) expertise to install, maintain, secure, and optimize enterprise Hadoop/Cloudera data platforms. The role requires deep expertise across cluster administration, automation, containerization, platform security, observability, and collaboration with data engineering teams to ensure reliable and high-performing data services.

Location:

Bengaluru, India

Experience:

8+ Years

Key Responsibilities

Cloudera & Data Platform Administration

• Provide hands-on Tier-2 production support for Cloudera Data Platform (CDP).
• Install, configure, maintain, and optimize the Cloudera environment.
• Install and maintain the latest supported CDP version, including CDP 7.1.9 or later and associated Cloudera Data Services.
• Support ongoing platform maintenance and operational stability.

Security & Compliance

• Implement and maintain security requirements for CDP.
• Automate security and regulatory checks.
• Support secure operation of the enterprise data platform.

Performance Engineering

• Optimize Cloudera cluster performance.
• Work closely with data engineering teams to optimize Spark job performance.
• Troubleshoot:
• Spark jobs
• Hive jobs
• YARN jobs
• Drive platform optimization and service reliability.

Containerization

• Implement CDP containerization with data services.
• Work with container technologies and enterprise container-management platforms.
• Support Kubernetes/OpenShift-based environments.

Automation & CI/CD

• Build automated pipelines and workflows for data projects.
• Use tools such as:
• GitHub Actions
• Jenkins
• ArgoCD
• Similar CI/CD technologies
• Python-based automation is preferred.
• Drive automated patching and platform observability.

Required Skills & Experience

• 8+ years of relevant experience, with the source specifically calling for 10+ years of hands-on Tier-2 production experience with Cloudera.
• Strong Cloudera/CDP administration experience.
• Experience developing automation using:
• Shell
• Terraform
• Groovy
• Ansible
• Experience with containerization technologies including:
• Podman
• Docker
• OCP
• Experience with container management using:
• Kubernetes
• OpenShift
• Strong Linux operating system and networking fundamentals.
• Proficiency with Git and Agile engineering practices.