Senior Platform Data Engineer (Databricks)

Anblicks; Hyderabad; On-site; INR 4,500,000 - 7,000,000; Full time

Benefits offered by this job

Databricks certification

Cloudera certification

AWS certification

Job summary

Anblicks is seeking a senior Architect to design and operate large-scale distributed data platforms using Databricks and Spark in Hyderabad. You will own end-to-end platform administration and drive production excellence.

You will optimize workloads, implement Infrastructure as Code, lead incident responses, and set performance standards across multi-cloud environments (AWS/Azure/GCP) and on-prem components. Strong governance and security practices are expected.

Qualifications

• 6+ years of data or cloud engineering experience.
• 2+ years operating Databricks in production.
• 2+ years working with Spark.
• 2+ years implementing Infrastructure as Code.
• Production ownership experience.
• End-to-end administration, support, and maintenance of enterprise data platforms including Databricks and Cloudera (CDP/CDH).
• Manages Databricks workspaces, compute, Unity Catalog, security, storage integrations, and CI/CD, with Cloudera services like HDFS, YARN, Hive, Impala, Spark, Oozie, Ranger, Atlas, Kerberos, TLS.
• Performs platform monitoring, incident troubleshooting, Spark job analysis, log investigation, disk space management, Linux operations, performance tuning, upgrades, automation, capacity planning, and high availability across cloud and on-prem environments.
• On-Call Support

Responsibilities

• Architect and operate large-scale distributed data platforms using Databricks and Spark.
• Optimize Spark and Delta Lake workloads.
• Implement Infrastructure as Code.
• Lead incident response.
• Drive performance standards.

Skills

6+ years data or cloud engineering

Production ownership experience

On-Call Support

Tools

Databricks

Spark

Terraform

Ansible

Chef

CI/CD

HDFS

YARN

Job description

Architect and operate large-scale distributed data platforms using Databricks and Spark.

What You Will Do

• Optimize Spark and Delta Lake workloads


• Implement Infrastructure as Code


• Lead incident response


• Drive performance standards

What You Will Have

• 6+ years of data or cloud engineering experience


• 2+ years operating Databricks in production


• 2+ years working with Spark


• 2+ years implementing Infrastructure as Code


• Production ownership experience


• Responsible for the end-to-end administration, support, and maintenance of enterprise data platforms, including Databricks and Cloudera (CDP/CDH).


• Manages Databricks workspaces, compute, Unity Catalog, security, storage integrations, and CI/CD, while supporting Cloudera services such as HDFS, YARN, Hive, Impala, Spark, Oozie, Ranger, Atlas, Kerberos, and TLS.


• Performs platform monitoring, incident troubleshooting, Spark job analysis, log investigation, disk space management, Linux-based operational support (services, processes, file systems, permissions, and logs), performance tuning, upgrades, automation (Ansible/chef/Terraform), capacity planning, and ensures high availability, security, governance, and operational excellence across AWS, Azure, GCP and on-perm Linux environments.


• On-Call Support

What Will Set You Apart

• Certifications: Databricks, Cloudera, AWS