Senior Platform Data Engineer

Anblicks; Hyderabad; On-site; INR 2,800,000 - 5,200,000; Full time

Job summary

Anblicks seeks a senior data engineer to architect and operate large-scale distributed data platforms using Databricks and Spark in Hyderabad. You will optimize Spark and Delta Lake workloads, implement Infrastructure as Code, and lead incident response to ensure high availability.

You will own end-to-end administration of enterprise data platforms, manage Databricks workspaces, security, and CI/CD, and coordinate across AWS, Azure, GCP, and on-prem environments. On-call support required.

Qualifications

• 8+ years in data or cloud engineering
• 2+ years operating Databricks in production
• 2+ years working with Spark
• 2+ years implementing Infrastructure as Code
• Production ownership experience
• End-to-end administration of enterprise data platforms (Databricks, Cloudera CDP/CDH)
• On-Call support and incident handling
• Platform monitoring, performance tuning and upgrades

Responsibilities

• Architect and operate large-scale distributed data platforms
• Optimize Spark and Delta Lake workloads
• Implement Infrastructure as Code
• Lead incident response
• Drive performance standards
• Manage Databricks workspaces, compute, Unity Catalog, security, storage integrations, and CI/CD
• Support Cloudera services (HDFS, YARN, Hive, Impala, Spark, Oozie, Ranger, Atlas, Kerberos, TLS) across multi-cloud and on-prem environments

Skills

Data engineering

Databricks prod

Spark

Infrastructure as Code

Production ownership

Platform monitoring

On-call support

Performance tuning

Tools

Databricks

Spark

Terraform

Ansible

Chef

CI/CD

Cloudera CDP/CDH

Job description

The Role

• Architect and operate large-scale distributed data platforms using Databricks and Spark.

What You Will Do

• Optimize Spark and Delta Lake workloads
• Implement Infrastructure as Code
• Lead incident response
• Drive performance standards

What You Will Have

• 8+ years of data or cloud engineering experience
• 2+ years operating Databricks in production
• 2+ years working with Spark
• 2+ years implementing Infrastructure as Code
• Production ownership experience
• Responsible for the end-to-end administration, support, and maintenance of enterprise data platforms, including Databricks and Cloudera (CDP/CDH).
• Manages Databricks workspaces, compute, Unity Catalog, security, storage integrations, and CI/CD, while supporting Cloudera services such as HDFS, YARN, Hive, Impala, Spark, Oozie, Ranger, Atlas, Kerberos, and TLS.
• Performs platform monitoring, incident troubleshooting, Spark job analysis, log investigation, disk space management, Linux-based operational support (services, processes, file systems, permissions, and logs), performance tuning, upgrades, automation (Ansible/chef/Terraform), capacity planning, and ensures high availability, security, governance, and operational excellence across AWS, Azure, GCP and on-perm Linux environments.
• On-Call Support

What Will Set You Apart

• Certifications: Databricks, Cloudera, AWS