Databricks Platform Engineer

Zorba Consulting India; Mumbai; On-site; INR 2,500,000 - 4,200,000; Full time

Job summary

Zorba Consulting India is seeking a hands-on Databricks Platform Engineer to provide offshore operational support and platform administration for an enterprise Databricks Lakehouse on AWS. You will work with onshore engineers to ensure platform reliability, governance, and cost optimization.

The role emphasizes incident management, security, and automation, using Terraform, Databricks REST APIs, Python scripting, and CLI to support deployments, configurations, and platform improvements.

Qualifications

• 3+ years of hands-on experience administering and supporting Databricks in a production environment.
• Experience with enterprise-scale Databricks environments on AWS.
• AWS certification and Databricks certification are required.

Responsibilities

• Provide Databricks platform administration and production operational support.
• Monitor platform cost, usage, capacity, performance, and resource utilization.
• Identify opportunities for Databricks and AWS cost optimization.
• Implement platform architecture, engineering standards, security controls, and operational processes.
• Monitor platform health and respond to incidents and operational alerts.
• Troubleshoot and restore platform services across Databricks, AWS, networking, security, compute, and workloads.
• Develop automation using Terraform, Databricks REST APIs, CLI, Python/Shell scripting.
• Support platform deployments, configuration changes, environment promotion, and release activities.
• Administer Databricks workspaces, users, groups, permissions, compute, clusters, policies, and platform configurations.
• Manage Unity Catalog, including catalogs, schemas, external locations, storage credentials, grants, and access governance.
• Configure and manage compute policies and job policies.
• Support AWS-integrated Databricks services and troubleshoot connectivity and security issues.
• Assist Data Engineering and Data Science teams with platform-related issues, pipeline failures, compute problems, access issues, and workload troubleshooting.
• Perform initial investigation of complex platform issues and escalate when required.
• Participate in POC initiatives and evaluate emerging Databricks capabilities.
• Create and maintain runbooks, operational procedures, troubleshooting guides, and platform documentation.
• Provide effective shift handoffs and offshore/onshore operational coverage.
• Continuously identify opportunities to improve platform reliability, automation, and support processes.

Skills

Databricks Admin

AWS Knowledge

Terraform

Databricks REST API

Python scripting

AWS IAM

Networking & Security

Unity Catalog

Incident management

Tools

Terraform

Job description

We are seeking a hands‑on Databricks Platform Engineer to provide offshore operational support, platform administration, automation, monitoring, and continuous improvement for an enterprise Databricks Lakehouse platform on AWS.

The engineer will work closely with the onshore Senior Data Platform Engineer while independently managing assigned platform engineering and operational responsibilities. The role focuses on platform reliability, automation, incident management, security, governance, cost optimization, and supporting Data Engineering and Data Science teams.

Key Responsibilities

• Provide day‑to‑day Databricks platform administration and production operational support.
• Monitor platform cost, usage, capacity, performance, and resource utilization.
• Identify and recommend opportunities for Databricks and AWS cost optimization.
• Implement approved platform architecture, engineering standards, security controls, and operational processes.
• Monitor platform health and respond to incidents, service requests, and operational alerts.
• Troubleshoot and restore platform services across Databricks, AWS, networking, security, compute, and workloads.
• Develop and maintain automation using Terraform, Databricks REST APIs, CLI, Python/Shell scripting, and other appropriate tools.
• Support platform deployments, configuration changes, environment promotion, and release activities.
• Administer Databricks workspaces, users, groups, permissions, compute, clusters, policies, and platform configurations.
• Manage Unity Catalog, including catalogs, schemas, external locations, storage credentials, grants, and access governance.
• Configure and manage compute policies and job policies.
• Support AWS‑integrated Databricks services and troubleshoot connectivity and security issues.
• Assist Data Engineering and Data Science teams with platform‑related issues, pipeline failures, compute problems, access issues, and workload troubleshooting.
• Perform initial investigation of complex platform issues and escalates to the appropriate engineering teams when required.
• Participate in proof‑of‑concept (POC) initiatives and evaluate emerging Databricks capabilities.
• Create and maintain runbooks, operational procedures, troubleshooting guides, and platform documentation.
• Provide effective shift handoffs and offshore/onshore operational coverage.
• Continuously identify opportunities to improve platform reliability, automation, operational efficiency, and support processes.

Required Qualifications

• 3+ years of hands‑on experience administering and supporting Databricks in a production environment.
• Strong experience supporting enterprise‑scale Databricks environments on AWS.
• AWS certification and Databricks certification are required.
• Strong hands‑on knowledge of:
• Databricks Account and Workspace Administration
• Unity Catalog and access governance
• Compute and Job Policies
• Databricks cost and usage optimization
• Databricks security and platform administration
• Databricks REST APIs and CLI
• Terraform-based infrastructure automation
• Strong AWS knowledge across:
• AWS IAM
• VPC, networking and security
• KMS
• DynamoDB
• MSK
• S3
• AWS PrivateLink
• CloudWatch
• Experience with Python/Shell scripting and automation.
• Good understanding of data pipelines, SQL, ETL processes, and production workload troubleshooting.
• Experience with monitoring, incident management, root‑cause investigation, and service restoration.
• Strong troubleshooting skills across platform, compute, networking, security, access, and workload layers.
• Good understanding of enterprise cloud security, governance, and operational best practices.
• Strong communication skills with the ability to work effectively with onshore engineering, Data Engineering, Data Science, and support teams.

Good to Have

• Experience with Databricks on AWS at enterprise scale.
• Experience with Databricks Jobs, Workflows, Delta Lake, SQL Warehouses, and serverless capabilities.
• Experience implementing Infrastructure as Code (IaC) using Terraform.
• Experience with CI/CD and DevOps tools such as Git, GitHub/GitLab, Azure DevOps, or Jenkins.
• Experience with Databricks cluster troubleshooting and Spark workload performance.
• Knowledge of AWS PrivateLink and secure Databricks networking architectures.
• Experience with platform observability and monitoring solutions.
• Experience participating in Databricks platform upgrades, migrations, and environment provisioning.
• Exposure to emerging Databricks features and platform POCs.