Senior Technology Engineer

Deloitte Shared Services India; Bengaluru, Delhi; Hybrid; INR 1,600,000 - 2,400,000; Full time

Job summary

Deloitte Shared Services India seeks a Senior Technology Engineer specializing in Hadoop/CDP and DevOps to manage production CDP clusters, ensure security, and optimize performance. You will automate provisioning, patching, and operations using Ansible, Terraform, Groovy, and Python, while building CI/CD pipelines and containerized data services.

The role requires hands-on experience with CDP versions, Spark/Hive/YARN, Linux, and modern DevOps tools.

Qualifications

• Strong hands-on production experience with Hadoop and CDP.
• Experience installing, configuring, upgrading, and maintaining CDP 7.1.7+ or latest CDP version.
• Experience with Cloudera Data Services 1.5.4 / 1.5.5 or later.
• Troubleshoot Spark, Hive, YARN, and Hadoop components.
• Security configuration and administration for CDP.
• Experience optimizing Hadoop/CDP clusters and Spark performance.
• Linux administration and networking fundamentals.
• Terraform and/or Infrastructure as Code expertise.
• CI/CD pipelines with Jenkins, GitHub Actions, or Argo CD.
• Git for source control and versioning.
• Docker, Podman, Kubernetes, OpenShift in production.
• Problem-solving and production troubleshooting skills.

Responsibilities

• Install, configure, upgrade, and maintain CDP clusters in production.
• Work with CDP 7.1.7+ and latest CDP versions.
• Configure Cloudera Data Services 1.5.x or later.
• Implement and maintain CDP security requirements.
• Monitor and optimize cluster health and performance.
• Troubleshoot Spark, Hive, YARN in the Hadoop ecosystem.
• Analyze Spark job performance with Data Engineers.
• Capacity planning, patching, upgrades, and maintenance.
• Support CDP containerization and containerized data services.
• Develop automation with Ansible, Shell, Terraform, Groovy, Python.
• Automate CDP provisioning, configuration, patching, and ops tasks.
• Build CI/CD pipelines with Jenkins, GitHub Actions, Argo CD.
• Automate security, compliance, and regulatory checks.
• Apply IaC and DevOps best practices to data platforms.
• Deploy container workloads with Kubernetes/OpenShift and manage observability.

Skills

Hadoop CDP

CDP cluster administration

Linux administration

Ansible

Terraform

CI/CD pipelines

Jenkins

Git

Kubernetes

OpenShift

Docker

Observability

Scripting (Python/Groovy)

Tools

CDP

Hadoop

Spark

YARN

Docker

Podman

Kubernetes

OpenShift

Git

Jenkins

Argo CD

Terraform

Ansible

Job description

Senior Technology Engineer Hadoop / CDP & DevOps

We are looking for a Senior Technology Engineer with strong hands-on experience in Hadoop and Cloudera Data Platform (CDP), complemented by DevOps and automation expertise. The role will focus primarily on CDP cluster administration, performance optimization, security, troubleshooting, automation, and containerized data services in a production environment.

The ideal candidate will work closely with Data Engineering and Platform teams to ensure highly available, secure, performant, and automated data platforms.

Key Responsibilities

Cloudera Data Platform / Hadoop

• Install, configure, upgrade, and maintain Cloudera Data Platform (CDP) clusters in production environments.
• Work with CDP 7.1.7+ and/or the latest available CDP versions.
• Configure and maintain Cloudera Data Services 1.5.4 / 1.5.5 or later.
• Implement and maintain CDP security requirements, including authentication, authorization, encryption, and security controls.
• Monitor and optimize cluster health, capacity, resource utilization, and overall performance.
• Troubleshoot complex issues across Spark, Hive, YARN, and Hadoop ecosystem components.
• Analyze and resolve Spark job performance issues in collaboration with Data Engineers.
• Perform cluster capacity planning, tuning, patching, upgrades, and maintenance activities.
• Support CDP containerization and containerized data services.

DevOps & Automation

• Develop and maintain automation using Ansible, Shell scripting, Terraform, Groovy, and Python.
• Automate CDP provisioning, configuration, patching, maintenance, and operational tasks.
• Build and maintain CI/CD pipelines and automated workflows for data and analytics projects.
• Work with tools such as GitHub Actions, Jenkins, and Argo CD.
• Automate security, compliance, and regulatory checks.
• Implement automated patching and configuration management processes.
• Apply Infrastructure as Code and DevOps best practices to data-platform environments.

Containerization & Platform Engineering

• Work with container technologies including Docker and Podman.
• Deploy and manage containerized workloads on Kubernetes and OpenShift (OCP).
• Support integration of CDP data services with containerized environments.
• Troubleshoot container, orchestration, networking, and platform-level issues.

Observability & Operations

• Implement and maintain platform monitoring, logging, alerting, and observability solutions.
• Establish operational dashboards and health checks for Hadoop/CDP environments.
• Proactively identify performance, capacity, reliability, and availability issues.
• Participate in production incident troubleshooting and root-cause analysis.

Required Technical Skills

Must-Have

• Strong hands-on production experience with Hadoop and Cloudera Data Platform (CDP).
• Experience installing, configuring, upgrading, and maintaining CDP 7.1.7+ or the latest CDP version.
• Experience with Cloudera Data Services 1.5.4 / 1.5.5 or later.
• Strong troubleshooting experience with:
• Apache Spark
• Hive
• YARN
• Hadoop
• Experience with CDP security configuration and administration.
• Experience optimizing Hadoop/CDP cluster and Spark job performance.
• Strong Linux/Unix administration skills.
• Good understanding of Linux networking fundamentals.
• Hands-on experience with Ansible and Shell scripting.
• Experience with Terraform and/or Infrastructure as Code.
• Experience developing or maintaining CI/CD pipelines using tools such as Jenkins, GitHub Actions, or Argo CD.
• Experience with Git/source control.
• Hands-on experience with Docker, Podman, Kubernetes, and/or OpenShift.
• Strong production troubleshooting and problem-solving skills.

Preferred

• Python development/scripting experience.
• Groovy scripting experience.
• Experience automating security and regulatory compliance checks.
• Experience with automated patching and configuration management.
• Experience implementing observability for data platforms.
• Experience working in Agile engineering environments.
• Experience collaborating with Data Engineering teams on Spark and analytics workload optimization.

DevOps / Engineering Practices

• Follow Agile engineering practices, including code reviews, CI/CD, automated testing, and continuous improvement.
• Develop reusable automation and operational tooling.
• Maintain infrastructure and application configurations through version control.
• Promote automation-first approaches to platform operations.
• Contribute to platform standards, documentation, and operational best practices.

Ideal Candidate Profile

The ideal candidate is a Hadoop/CDP-focused technology engineer who combines strong production platform administration skills with modern DevOps capabilities. They should be comfortable troubleshooting complex data-platform issues, automating repetitive operational activities, working with container platforms, and partnering with Data Engineering teams to improve reliability and performance.