Senior ClickHouse Database Engineer

Persistent Systems Limited; Pune District; On-site; INR 4,000,000 - 6,000,000; Full time

Benefits offered by this job

Hybrid work culture

Flexible hours

Long Service awards

Insurance coverage

Job summary

Persistent Systems Limited is seeking a Senior ClickHouse Database Engineer to own our analytics platform, ensuring high availability and performance in geo-distributed environments.

You will lead upgrades, topology design, and security controls while optimizing queries and managing backups across production and staging systems. A strong emphasis on collaboration with cross-functional teams and mentoring is required.

Qualifications

• 10-15 years of experience in database/platform engineering or related field.
• Extensive hands-on experience administering ClickHouse in large-scale production environments.
• Deep understanding of ClickHouse internals and MergeTree storage engines.

Responsibilities

• Administer and manage enterprise-scale ClickHouse clusters across production and non-production environments.
• Plan and execute ClickHouse version upgrades, compatibility assessments, rollback strategies, and post-upgrade validations.
• Design and maintain cluster topologies, shard configurations, replica management, and node allocation strategies.
• Lead ZooKeeper to ClickHouse Keeper migration initiatives and ongoing Keeper administration.
• Configure and manage backup and recovery solutions using S3-compatible storage platforms.
• Develop and execute disaster recovery, failover, and business continuity strategies.
• Implement and maintain ClickHouse security controls including TLS, RBAC, network isolation, and user-role management.
• Design, optimize, and evolve materialized view architectures using AggregatingMergeTree and related engines.
• Manage geo-redundant WAN replication across distributed deployments and availability zones.
• Perform database performance tuning and workload optimization for large-scale analytical queries.
• Monitor cluster health, replication status, storage utilization, and operational KPIs.
• Build and maintain observability dashboards using Grafana and Prometheus.
• Troubleshoot production issues using ClickHouse system tables and diagnostic tools.
• Manage maintenance activities including part merges, TTL policies, and storage lifecycle operations.
• Support Kafka-to-ClickHouse ingestion pipelines and resolve end-to-end data flow issues.
• Investigate consumer lag, ingestion bottlenecks, serialization failures, and pipeline performance issues.
• Develop automation scripts for maintenance, health monitoring, backups, and operational workflows.
• Collaborate with Data Engineering, Platform Engineering, DevOps, Infrastructure, and Security teams.
• Participate in architecture reviews, production readiness assessments, and capacity planning exercises.
• Mentor junior engineers and contribute to operational runbooks, documentation, and best practices.

Skills

ClickHouse

Database administration

Performance tuning

Security

Shell scripting

Automation

Kubernetes

Kafka

Observability

Terraform

Tools

Kubernetes

Kafka

Prometheus

Grafana

Helm

Longhorn

Terraform

Job description

We are an AI-led, platform-driven Digital Engineering and Enterprise Modernization partner, combining deep technical expertise and industry experience to help our clients anticipate what's next. Our offerings and proven solutions create a unique competitive advantage for our clients by giving them the power to see beyond and rise above. We work with many industry-leading organizations across the world, including 20 Fortune 50 companies and 4 of the 5 top banks in both the US and India, and numerous innovators across the healthcare ecosystem.

We are looking for a highly experienced Senior ClickHouse Database Engineer to own and manage the ClickHouse analytics database platform that powers our real-time network analytics products. This platform serves as the primary data store for analytics counters, KPI aggregations, reporting workloads, and interactive dashboards operating at enterprise scale. The ideal candidate will possess deep expertise in ClickHouse internals, database administration, Kubernetes-based deployments, Kafka integration, performance tuning, replication, security, and production troubleshooting. You will be responsible for the complete lifecycle management of the ClickHouse platform, ensuring scalability, reliability, high availability, and operational excellence across geo-distributed environments.

• Location: Pune
• Experience: 10 to 15 Years
• Job Type: Full-Time Employment

What You'll Do:

• Administer and manage enterprise-scale ClickHouse clusters across production and non-production environments.
• Plan and execute ClickHouse version upgrades, compatibility assessments, rollback strategies, and post-upgrade validations.
• Design and maintain cluster topologies, shard configurations, replica management, and node allocation strategies.
• Lead ZooKeeper to ClickHouse Keeper migration initiatives and ongoing Keeper administration.
• Configure and manage backup and recovery solutions using S3-compatible storage platforms.
• Develop and execute disaster recovery, failover, and business continuity strategies.
• Implement and maintain ClickHouse security controls including TLS, RBAC, network isolation, and user-role management.
• Design, optimize, and evolve materialized view architectures using AggregatingMergeTree and related engines.
• Manage geo-redundant WAN replication across distributed deployments and availability zones.
• Perform database performance tuning and workload optimization for large-scale analytical queries.
• Monitor cluster health, replication status, storage utilization, and operational KPIs.
• Build and maintain observability dashboards using Grafana and Prometheus.
• Troubleshoot production issues using ClickHouse system tables and diagnostic tools.
• Manage maintenance activities including part merges, TTL policies, and storage lifecycle operations.
• Support Kafka-to-ClickHouse ingestion pipelines and resolve end-to-end data flow issues.
• Investigate consumer lag, ingestion bottlenecks, serialization failures, and pipeline performance issues.
• Develop automation scripts for maintenance, health monitoring, backups, and operational workflows.
• Collaborate with Data Engineering, Platform Engineering, DevOps, Infrastructure, and Security teams.
• Participate in architecture reviews, production readiness assessments, and capacity planning exercises.
• Mentor junior engineers and contribute to operational runbooks, documentation, and best practices.

Expertise You'll Bring:

• 10-15 years of overall database, infrastructure, or platform engineering experience.
• Extensive hands-on experience administering ClickHouse in large-scale production environments.
• Deep understanding of ClickHouse internals and MergeTree storage engines including ReplicatedMergeTree, AggregatingMergeTree, SummingMergeTree.
• Expertise in columnar storage architecture, part lifecycle management, and merge processing.
• Experience managing ClickHouse clusters with multiple shards and replicas.
• Strong understanding of distributed DDL operations and cluster configuration management.
• Proven experience performing ClickHouse upgrades, migrations, rollback planning, and version lifecycle management.
• Experience implementing backup, restore, and disaster recovery solutions for ClickHouse clusters.
• Strong proficiency in ClickHouse SQL, query optimization, execution plan analysis, and performance tuning.
• Experience working with system tables such as system.parts, system.merges, system.query_log, system.replication_queue, system.errors.
• Experience managing ClickHouse Keeper and ZooKeeper environments.
• Strong understanding of ClickHouse security architecture, access control, encryption, and auditing.
• Strong hands-on experience managing stateful workloads on Kubernetes.
• Expertise with StatefulSets, Persistent Volumes, storage classes, and database workloads.
• Experience with Longhorn, local-path storage, and storage migration strategies.
• Strong experience deploying and maintaining applications using Helm.
• Knowledge of Helm chart customization, upgrades, and migration activities.
• Experience with TLS certificate lifecycle management using cert-manager or equivalent platforms.
• Strong knowledge of infrastructure automation and operational best practices.
• Strong understanding of Apache Kafka architecture and operations.
• Experience with topics, partitions, consumer groups, replication, and offset management.
• Ability to troubleshoot Kafka-to-ClickHouse ingestion pipelines.
• Experience identifying and resolving Consumer lag issues, Schema compatibility problems, Deserialization failures, Dead-letter queue scenarios, Offset recovery situations.
• Knowledge of real-time data ingestion and event-driven analytics platforms.
• Experience correlating database bottlenecks with Kafka ingestion performance.
• Hands-on experience with Prometheus and Grafana monitoring solutions.
• Experience building dashboards for replication lag, query latency, merge backlogs, and cluster health metrics.
• Strong troubleshooting and production incident management skills.
• Familiarity with centralized logging solutions such as Loki, ELK, or equivalent platforms.
• Ability to perform root-cause analysis and implement preventive measures.
• Strong scripting experience using Bash and Shell scripting.
• Experience automating maintenance and operational workflows.
• Working knowledge of Java for ClickHouse client integrations and upgrade assessments.
• Familiarity with Git and source-code management practices.
• Experience working with CI/CD pipelines and Infrastructure-as-Code methodologies.
• Exposure to Terraform and automation-driven deployment models.
• Strong analytical and problem-solving capabilities.
• Experience operating mission-critical production platforms.
• Excellent communication and stakeholder management skills.
• Ability to collaborate effectively with cross-functional teams.
• Strong mentoring and technical leadership abilities.
• Proven experience driving operational excellence and platform reliability initiatives.
• Competitive salary and benefits package
• Culture focused on talent development with quarterly growth opportunities and company-sponsored higher education and certifications
• Opportunity to work with cutting-edge technologies
• Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
• Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents

Values-Driven, People-Centric & Inclusive Work Environment:

Persistent is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds.

• We support hybrid work and flexible hours to fit diverse lifestyles.
• Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.
• If you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment

"Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind."