Lead Software Engineer - Cloudera Data Platform - IT (Information Technology) - Bangalore, India

Societe Generale SA; Bengaluru; On-site; INR 1,800,000 - 3,000,000; Full time

Job summary

Société Générale invites a Senior Cloudera Data Platform, NiFi & Airflow Administrator to join our enterprise data platform operations in Bengaluru. You will own end-to-end administration of Cloudera Data Platform environments, Apache NiFi data flows, and Apache Airflow orchestration, across production and non-production environments, ensuring security, reliability and performance.

The role requires strong Linux skills, experience with distributed data processing, capacity planning, incident

Qualifications

• 5+ years of experience in senior data platform roles.
• Strong expertise with Cloudera Data Platform, NiFi and Airflow.
• Proficient Linux administration and security practices.

Responsibilities

• Operate and maintain enterprise data platforms.
• Ensure secure, reliable operation of Cloudera, NiFi and Airflow across environments.
• Design high-availability concepts and disaster recovery plans.
• Troubleshoot complex issues across distributed services and infrastructure.
• Support analytics and data science teams with workload design and production readiness.
• Manage platform configuration, access control, and security integrations.
• Perform capacity planning and performance optimization; monitor KPI and throughput.
• Develop automation and operational tooling for deployment, monitoring and alerts.
• Lead platform upgrades, migrations, and incident response activities.
• Maintain documentation, runbooks, and governance artifacts.

Skills

Cloudera Data Platform

Apache NiFi

Apache Airflow

Linux administration

Troubleshooting

Tools

Cloudera Manager

HDFS

YARN

Hive

Impala

Spark

Ranger

Atlas

Knox

ZooKeeper

Kerberos

Ozone

Iceberg

Job description

ResponsibilitiesDATA PLATFORMTECHNOLOGY / JOB DESCRIPTIONSenior Cloudera Data Platform,NiFi& Airflow AdministratorSenior / ExpertLevel | Enterprise Data Platform Operations

Short OverviewThe Senior Cloudera,NiFi& Airflow Administratoris responsible forthe administration, design, operation, reliability, security, and continuous improvement of enterprise data platform services. The role provides end-to-end administration of Cloudera Data Platform environments, ApacheNiFidata-flow services, and Apache Airflow orchestration services across production and non-production environments. The administrator is expert for distributed data processing, data engines, data ingestion, workflow orchestration, platform automation, observability, troubleshooting, capacity planning, lifecycle management, and operational governance in a mission-critical enterprise environment.The role also requires strong Linux administration skills, including system configuration, troubleshooting, performance tuning, networking, storage management, and security.

Overall Experience: 5+ years, senior / expert level

Key Responsibilities

• Operate andmaintainenterprise data platforms
• Ensure secure, reliable, and stable operation, availability, performance, and resilience of Cloudera, Apache NiFi, and Apache Airflow services across production and non-production environments, including platform security, access control, authentication, authorization, encryption, and compliance with enterprise security standards.
• Design and evolve platform architectureDefine and review high-availability concepts, disaster recovery arrangements, environment topology, service placement, integration patterns, configuration standards, naming conventions, and operational guidelines.
• Provide senior-level troubleshooting and incident supportAnalyze and resolve complex issues across distributed services, workloads, data flows, workflows, metadata, security, networking, operating systems, storage, databases, JVMs, and dependent infrastructure.
• Support application, analytics, data engineering and data science teamsConsult teams on solution onboarding, workload design, resource utilization, scheduling, data movement, dependency management, resilience and error handling, operational readiness, and production support requirements.
• Manage platform configuration, security, and access controlMaintain authentication, authorization, roles, policies, service accounts, certificates, TLS, secrets, network connectivity, encryption settings, and integrations with enterprise identity and security services.
• Perform capacity planning and performance optimizationMonitor workload concurrency, queues, memory, CPU, storage, network throughput, JVM behavior, NiFiback pressure, repository usage, Airflow scheduler performance, task throughput, metadata database health, and service saturation; recommend scaling and tuning actions.
• Support andmaintainnon-production and production environmentsProvide maintenance,refresh, patching, configuration alignment, access management, test support, troubleshooting, and lifecycle activities for DEV, TEST, SIT, UAT, PROD, and similar environments.
• Develop automation and operational toolingCreate and improve scripts, configuration templates, deployment automation, health checks, dashboards, alerts, reporting, and self-service procedures using infrastructure-as-code and CI/CD practices.
• Manage platform lifecycle and vulnerability remediationCoordinate version currency, upgrades, patches, vulnerability assessment, remediation plans, exception handling, risk acceptance, end-of-support tracking, and compliance evidence for platform components.
• Lead changes, upgrades, migrations, and recovery testsPlan and execute platform upgrades, migrations, configuration changes, certificate renewals, componentreplacements, backup and restore validation, failover exercises, and disaster recovery tests with controlled business impact.
• Ensure observability and operational reportingDefine monitoring standards, alert thresholds, service dashboards, availability views, incident trends, capacity reports, and operationalKPIs for the supported services.
• Contribute to service governance and complianceMaintain architecture records, technical documentation, runbooks, standards, configuration inventories, risk records, audit evidence, operational controls, and support procedures.
• Act as platform subject matter expertProvideexpert guidance on distributed data platforms, data ingestion, orchestration, resilience, security, platformselection, and operational best practices.
• Participate in on‑call, major incident, and problem managementSupport critical incidents, root cause analysis, post‑incident reviews, permanent corrective actions, and knowledge sharing across operations, engineering, architecture, and application teams.

Cloudera‑Specific Responsibilities

• Administer Cloudera Data Platform services and supporting components, including Cloudera Manager, HDFS, YARN, Hive, Impala, Spark, Ranger, Atlas, Knox,ZooKeeper, Kerberos, Ozone , OTF Iceberg and related platformservices wheredeployed.
• Manage service configuration, role placement, parcels and runtime versions, host maintenance, commissioning and decommissioning, rolling restarts, health checks, configuration validation, and environment consistency.
• Monitor and troubleshoot cluster health, HDFS and Ozone capacity, replication, under-replicated data, YARN queues, workload contention, HiveMetastore, Impala admission control, Spark execution, service logs, JVM resources, and host-level dependencies.
• Administer data-platform security, including Kerberos integration, Ranger policies, Knox access, TLS certificates,keytabs, service identities, encryption‑related configuration, and audit logging.
• Support backup, restore, replication, disaster recovery, data retention, metadata protection, and recoverytesting forplatform services and configuration.
• Plan cluster expansion, operating system and Java compatibility changes, platform upgrades, service migrations, storage evolution, and decommissioning of obsolete components.

ApacheNiFi‑Specific Responsibilities

• Administer clustered ApacheNiFienvironments, including nodes, process groups, processors, controller services, parameter contexts, reporting tasks, connection pools, repositories, and registry integration.
• Operate and troubleshoot data flows, queues, back pressure, provenance, bulletins, failed transactions, state management,priorities, concurrent tasks, scheduling, record processors, and interactions with source and target systems.
• Define standards for reusable flow design, version control, promotion between environments, parameterization, secret handling, naming, documentation, error routing, retry logic, dead‑letter patterns, and operational ownership.
• ManageNiFisecurity, including TLS, certificates, authentication, authorization policies, service accounts, proxy configuration, sensitive properties, and integration with enterprise identity services.
• Plan and executeNiFiupgrades, node replacement, cluster expansion, repository maintenance, certificate renewal, flow migration, backup and recovery activities, and resilience testing.
• Monitor flow throughput, queue growth, back pressure, repositoryutilization, JVM resources, garbage collection, thread usage, connection status, and end‑to‑end delivery behavior.

ApacheAirflow‑Specific Responsibilities

• Administer Apache Airflow environments and core components, including schedulers, webservers,triggers, workers, executors, metadata databases, DAG processing, connections, variables, pools, queues, plugins, and providers.
• Operate and troubleshoot DAG parsing, task scheduling, retries, dependencies, sensors, deferrable operators, worker capacity, queue behavior, executor performance, metadata database contention, failed tasks, and scheduler availability.
• Define standards for DAG structure, deployment, version control, environment promotion, secrets, connections, variables, pools, concurrency, retries, alerting, ownership, documentation, and operational readiness.
• Manage Airflow security, including authentication, role‑based access control,secretsbackends, service identities, TLS, API access, auditability, and integration with enterprise identity and security services.
• Plan and execute Airflow upgrades, provider updates, executor or worker changes, metadata database maintenance, DAG migration, backup and restore validation, and high‑availability improvements.
• Monitor scheduler heartbeat, DAG processing time, task latency, queue depth, workerutilization, executor state,triggererhealth, metadata database performance, and availability of dependent services.

Key Required Knowledge and Skills

• Deep knowledge of distributed data platform operations, data ingestion, batch and streaming integration, workflow orchestration, scheduling, dependency management, metadata, and data lifecycle concepts.
• Hands‑on production administration experience with Cloudera Data Platform, ApacheNiFi, and Apache Airflow.
• Strong troubleshooting capability acrossapplication,platform, JVM, database, operating system, network, storage, and security layers.
• Ability to analyze logs, metrics, traces, service states, resource queues, flow files, back pressure, provenance, DAG runs, task instances, scheduler behavior, and infrastructure dependencies.
• Ability to lead root cause analysis and define permanent corrective actions for recurringplatform, integration, workload, or orchestration issues.
• Experience with high availability, clustering, failover, disaster recovery, backup and restore, business continuity, and recovery testing.
• Strong understanding of security controls: Kerberos, TLS, certificates, LDAP/Active Directory integration, role‑based access control, Apache Ranger,secretsmanagement, encryption, and audit logging.
• Good scripting and automation skills using Bash, Python, Ansible, Git, CI/CD tooling, or equivalent technologies.
• Strong Linux administration skills and practical knowledge of JVM tuning,systemd, networking, DNS, load balancing, firewalls, storage, databases, and container platforms.
• Knowledge of IT service management processes, including incident, problem, change, release, configuration, vulnerability, lifecycle, and capacity management.
• Ability tomaintainclear technical documentation, runbooks, support models, architecture records, operational procedures, and audit evidence.
• Strong communicationand stakeholder management skills, including the ability to coordinate technical activities during incidents, changes, upgrades, migrations, releases, and service improvements.

Profile requiredKey Required Technologies Cloudera Data Platform Cloudera Manager, CDP Private Cloud Base, HDFS, Ozone, OTF Iceberg, YARN, Hive, Impala, Spark, Ranger, Atlas, Knox,ZooKeeper, Kerberos, HiveMetastore, resource management, workload governance,backupand disaster recovery.

Automation and platform engineering

• Python, Bash, Ansible, Git, CI/CD pipelines, infrastructure as code, REST APIs, configuration management, container images, Kubernetes, Helm, and artifact repositories.

Databases and integration

• SQL, JDBC/ODBC, relational databases, object storage, Kafka or comparable messaging services, SFTP, REST services, and enterprise integration patterns.

Observability

• Prometheus, Grafana, ELK/OpenSearch, Cloudera Manager monitoring, Airflow metrics,NiFireporting tasks, or comparable enterprise tools.

Infrastructure and security

• Linux, Java/JVM, Python runtimes, PostgreSQL or comparable metadata databases, networking, DNS, TLS, certificates, load balancing, nginx, firewalls, proxies, storage, identity management, andsecretsmanagement.

ITSM and collaboration

• SMAX, Jira, Confluence, Azure DevOps, Teams, or comparable tools used for incident, change, problem, release, and documentation management.

Service Availability

Business days (Monday–Friday), 8:30 AM–5:00 PM Central European Time. Participation in planned maintenance windows, major incident support, release support, lifecycle activities, vulnerability remediation, security patching, disaster recovery tests, and on‑call or standby coverage may berequiredfor production and critical non-production services.

Why join us

We are committed to creating a diverse environment and are proud to be an equal opportunity employer. All qualified applicants receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, or veteran status.

Business insight

At Société Générale, we are convinced that people are drivers of change, and that the world of tomorrow will be shaped by all their initiatives, from the smallest to the most ambitious. Whether you’re joining us for a period of months, years or your entire career, together we can have a positive impact on the future. Creating, daring, innovating, and taking action are part of our DNA. If you too want to be directly involved, grow in a stimulating and caring environment, feel useful on a dailybasis and develop or strengthen your expertise, you will feel right at home with us!

Still hesitating?

You should know that our employees can dedicate several days per year to solidarity actions during their working hours, including sponsoring people struggling with their orientation or professional integration, participating in the financial education of young apprentices, and sharing their skills with charities. There are many ways to get involved.

We are committed to support accelerating our Group’s ESG strategy by implementing ESG principles in all our activities and policies. They are translated in our business activity (ESG assessment, reporting, project management or IT activities), our work environment and in our responsible practices for environment protection.