Senior Platform Data Engineer (Databricks)

Anblicks; Hyderabad; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

Anblicks is looking for an Architect to design and operate large-scale distributed data platforms using Databricks and Spark. You will own end-to-end administration, support, and maintenance of enterprise data platforms and drive reliability across multi-cloud environments.

You will optimize workloads, implement infrastructure as code, lead incident response, and ensure security, governance, and performance standards.

Qualifications

• 8+ years of data or cloud engineering experience.
• 2+ years operating Databricks in production.
• 2+ years working with Spark.
• 2+ years implementing Infrastructure as Code.
• Production ownership experience.
• End-to-end administration of enterprise data platforms including Databricks and Cloudera.

Responsibilities

• Optimize Spark and Delta Lake workloads.
• Implement Infrastructure as Code.
• Lead incident response.
• Drive performance standards.

Skills

Production ownership

Incident response

Performance tuning

Platform monitoring

Linux

Tools

Databricks

Spark

Terraform

Ansible

Chef

Cloudera

HDFS

YARN

Hive

Oozie

Ranger

Atlas

Kerberos

TLS

Job description

Architect and operate large-scale distributed data platforms using Databricks and Spark.

What You Will Do

• Optimize Spark and Delta Lake workloads
• Implement Infrastructure as Code
• Lead incident response
• Drive performance standards

What You Will Have

• 8+ years of data or cloud engineering experience
• 2+ years operating Databricks in production
• 2+ years working with Spark
• 2+ years implementing Infrastructure as Code
• Production ownership experience
• Responsible for the end-to-end administration, support, and maintenance of enterprise data platforms, including Databricks and Cloudera (CDP/CDH).
• Manages Databricks workspaces, compute, Unity Catalog, security, storage integrations, and CI/CD, while supporting Cloudera services such as HDFS, YARN, Hive, Impala, Spark, Oozie, Ranger, Atlas, Kerberos, and TLS.
• Performs platform monitoring, incident troubleshooting, Spark job analysis, log investigation, disk space management, Linux-based operational support (services, processes, file systems, permissions, and logs), performance tuning, upgrades, automation (Ansible/chef/Terraform), capacity planning, and ensures high availability, security, governance, and operational excellence across AWS, Azure, GCP and on-perm Linux environments.
• On-Call Support

What Will Set You Apart

• Certifications: Databricks, Cloudera, AWS