Senior ClickHouse Database Engineer

Persistent; Pune District; On-site; INR 500,000 - 900,000; Full time

Benefits offered by this job

Competitive salary

Hybrid work

Health insurance

Career development

Annual health check-ups

Education sponsorship

Long-term incentives

Job summary

Persistent is seeking a Senior ClickHouse Database Engineer in Pune to own and manage a large-scale analytics platform. You will lead deployments, upgrades, and topology design, ensuring high availability and performance across geo-distributed environments.

The role demands deep expertise in ClickHouse internals, Keeper administration, backup strategies, and security, with extensive experience in Kubernetes-based deployments and data pipelines with Kafka.

Qualifications

• 10–15 years of database, infra, or platform engineering experience.
• Extensive hands-on experience administering ClickHouse in large-scale production environments.
• Deep understanding of ClickHouse internals and MergeTree engines (ReplicatedMergeTree, AggregatingMergeTree, SummingMergeTree).
• Experience with Kubernetes-based deployments and stateful workloads.

Responsibilities

• Administer and manage enterprise-scale ClickHouse clusters across prod and non-prod environments.
• Plan and execute ClickHouse upgrades, compatibility assessments, rollback strategies, and post-upgrade validations.
• Design and maintain cluster topologies, shard configurations, replica management, and node allocation strategies.
• Lead ZooKeeper to ClickHouse Keeper migration initiatives and Keeper administration.
• Configure and manage backup and recovery solutions using S3-compatible storage platforms.
• Develop and execute disaster recovery, failover, and business continuity strategies.
• Implement and maintain ClickHouse security controls including TLS, RBAC, network isolation, and user-role management.
• Design, optimize, and evolve materialized view architectures using AggregatingMergeTree and related engines.
• Manage geo-redundant WAN replication across distributed deployments and availability zones.
• Perform database performance tuning and workload optimization for large-scale analytical queries.
• Monitor cluster health, replication status, storage utilization, and operational KPIs.
• Build observability dashboards using Grafana and Prometheus.
• Troubleshoot production issues using ClickHouse system tables and diagnostic tools.
• Manage maintenance activities including part merges, TTL policies, and storage lifecycle operations.
• Support Kafka-to-ClickHouse ingestion pipelines and resolve end-to-end data flow issues.
• Investigate consumer lag, ingestion bottlenecks, serialization failures, and pipeline performance issues.
• Develop automation scripts for maintenance, health monitoring, backups, and operational workflows.
• Collaborate with Data Engineering, Platform Engineering, DevOps, Infrastructure, and Security teams.
• Participate in architecture reviews, production readiness assessments, and capacity planning exercises.
• Mentor junior engineers and contribute to runbooks and best practices.]
• COMPANY NAME: Persistent
• KEY POINTS: Senior ClickHouse expert, Kubernetes and Kafka integration, disaster recovery, security controls

Skills

ClickHouse

Kubernetes

Kafka

Performance tuning

Replication

RBAC

TLS

ZooKeeper

Prometheus

Grafana

Tools

ZooKeeper

ClickHouse Keeper

Helm

cert-manager

Longhorn

Terraform

Job description

About Position:

We are looking for a highly experienced Senior ClickHouse Database Engineer to own and manage the ClickHouse analytics database platform that powers our real-time network analytics products. This platform serves as the primary data store for analytics counters, KPI aggregations, reporting workloads, and interactive dashboards operating at enterprise scale. The ideal candidate will possess deep expertise in ClickHouse internals, database administration, Kubernetes-based deployments, Kafka integration, performance tuning, replication, security, and production troubleshooting. You will be responsible for the complete lifecycle management of the ClickHouse platform, ensuring scalability, reliability, high availability, and operational excellence across geo-distributed environments.

• Role: Senior ClickHouse Database Engineer


• Location: Pune


• Experience: 10 to 15 Years


• Job Type: Full-Time Employment

What You'll Do:

• Administer and manage enterprise-scale ClickHouse clusters across production and non-production environments.


• Plan and execute ClickHouse version upgrades, compatibility assessments, rollback strategies, and post-upgrade validations.


• Design and maintain cluster topologies, shard configurations, replica management, and node allocation strategies.


• Lead ZooKeeper to ClickHouse Keeper migration initiatives and ongoing Keeper administration.


• Configure and manage backup and recovery solutions using S3-compatible storage platforms.


• Develop and execute disaster recovery, failover, and business continuity strategies.


• Implement and maintain ClickHouse security controls including TLS, RBAC, network isolation, and user-role management.


• Design, optimize, and evolve materialized view architectures using AggregatingMergeTree and related engines.


• Manage geo-redundant WAN replication across distributed deployments and availability zones.


• Perform database performance tuning and workload optimization for large-scale analytical queries.


• Monitor cluster health, replication status, storage utilization, and operational KPIs.


• Build and maintain observability dashboards using Grafana and Prometheus.


• Troubleshoot production issues using ClickHouse system tables and diagnostic tools.


• Manage maintenance activities including part merges, TTL policies, and storage lifecycle operations.


• Support Kafka-to-ClickHouse ingestion pipelines and resolve end-to-end data flow issues.


• Investigate consumer lag, ingestion bottlenecks, serialization failures, and pipeline performance issues.


• Develop automation scripts for maintenance, health monitoring, backups, and operational workflows.


• Collaborate with Data Engineering, Platform Engineering, DevOps, Infrastructure, and Security teams.


• Participate in architecture reviews, production readiness assessments, and capacity planning exercises.


• Mentor junior engineers and contribute to operational runbooks, documentation, and best practices.

Expertise You'll Bring:

• 10-15 years of overall database, infrastructure, or platform engineering experience.


• Extensive hands-on experience administering ClickHouse in large-scale production environments.


• Deep understanding of ClickHouse internals and MergeTree storage engines including ReplicatedMergeTree, AggregatingMergeTree, SummingMergeTree.


• Expertise in columnar storage architecture, part lifecycle management, and merge processing.


• Experience managing ClickHouse clusters with multiple shards and replicas.


• Strong understanding of distributed DDL operations and cluster configuration management.


• Proven experience performing ClickHouse upgrades, migrations, rollback planning, and version lifecycle management.


• Experience implementing backup, restore, and disaster recovery solutions for ClickHouse clusters.


• Strong proficiency in ClickHouse SQL, query optimization, execution plan analysis, and performance tuning.


• Experience working with system tables such as system.parts, system.merges, system.query_log, system.replication_queue, system.errors.


• Experience managing ClickHouse Keeper and ZooKeeper environments.


• Strong understanding of ClickHouse security architecture, access control, encryption, and auditing.


• Strong hands-on experience managing stateful workloads on Kubernetes.


• Expertise with StatefulSets, Persistent Volumes, storage classes, and database workloads.


• Experience with Longhorn, local-path storage, and storage migration strategies.


• Strong experience deploying and maintaining applications using Helm.


• Knowledge of Helm chart customization, upgrades, and migration activities.


• Experience with TLS certificate lifecycle management using cert-manager or equivalent platforms.


• Strong knowledge of infrastructure automation and operational best practices.


• Strong understanding of Apache Kafka architecture and operations.


• Experience with topics, partitions, consumer groups, replication, and offset management.


• Ability to troubleshoot Kafka-to-ClickHouse ingestion pipelines.


• Experience identifying and resolving Consumer lag issues, Schema compatibility problems, Deserialization failures, Dead-letter queue scenarios, Offset recovery situations.


• Knowledge of real-time data ingestion and event-driven analytics platforms.


• Experience correlating database bottlenecks with Kafka ingestion performance.


• Hands-on experience with Prometheus and Grafana monitoring solutions.


• Experience building dashboards for replication lag, query latency, merge backlogs, and cluster health metrics.


• Strong troubleshooting and production incident management skills.


• Familiarity with centralized logging solutions such as Loki, ELK, or equivalent platforms.


• Ability to perform root-cause analysis and implement preventive measures.


• Strong scripting experience using Bash and Shell scripting.


• Experience automating maintenance and operational workflows.


• Working knowledge of Java for ClickHouse client integrations and upgrade assessments.


• Familiarity with Git and source-code management practices.


• Experience working with CI/CD pipelines and Infrastructure-as-Code methodologies.


• Exposure to Terraform and automation-driven deployment models.


• Strong analytical and problem-solving capabilities.


• Experience operating mission-critical production platforms.


• Excellent communication and stakeholder management skills.


• Ability to collaborate effectively with cross-functional teams.


• Strong mentoring and technical leadership abilities.


• Proven experience driving operational excellence and platform reliability initiatives.

Benefits:

• Competitive salary and benefits package


• Culture focused on talent development with quarterly growth opportunities and company-sponsored higher education and certifications


• Opportunity to work with cutting-edge technologies


• Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards


• Annual health check-ups


• Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents

Values-Driven, People-Centric & Inclusive Work Environment:

Persistent is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds.

• We support hybrid work and flexible hours to fit diverse lifestyles.


• Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.


• If you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment

Let's unleash your full potential at Persistent-persistent.com/careers

Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind.