Databricks Platform Engineer

Virtusa; Bengaluru Urban; On-site; INR 1,800,000 - 2,800,000; Full time

Job summary

Virtusa Bengaluru is seeking an offshore Platform Engineer to administer Databricks environments, including clusters, jobs and workflows for enterprise AI solutions.

You will manage Unity Catalog for data governance, implement CI/CD pipelines and enforce security and cost controls through cluster policies. The role also covers Spark Structured Streaming pipelines, reliable data ingestion from Kafka, and platform health monitoring with Azure Monitor and Dynatrace.

Qualifications

• 4+ years of Databricks/Spark experience.
• Proficiency in Terraform and Git-based CI/CD.
• Strong Python and SQL skills.
• Bachelor's degree required.
• Databricks certifications and ITIL familiarity preferred.

Responsibilities

• Manage Unity Catalog for data governance and access control.
• Implement CI/CD pipelines for Databricks workloads.
• Enforce security and cost controls via cluster policies.
• Oversee Spark Structured Streaming pipelines from Kafka.
• Monitor platform health with Azure Monitor and Dynatrace.
• Troubleshoot performance and scaling of Databricks environments.

Skills

Databricks/Spark

Terraform

Git-based CI/CD

Python

SQL

Education

Bachelor's degree

Tools

Azure Monitor

Dynatrace

Unity Catalog

Job description

"This offshore Platform Engineer role focuses on administering and operating Databricks environments, including clusters, jobs, and workflows, to support enterprise Agentic AI solutions."

"Key responsibilities include managing Unity Catalog for data governance, implementing CI/CD pipelines, and enforcing security and cost controls through cluster policies."

"The engineer will also oversee Spark Structured Streaming pipelines, ensuring reliable data ingestion from messaging services like Kafka."

"Monitoring platform health via Azure Monitor and Dynatrace, alongside troubleshooting performance and scaling issues, is critical to the role."

"Candidates require 4+ years of Databricks/Spark experience, proficiency in Terraform and Git-based CI/CD, and strong skills in Python and SQL."

"The position demands a bachelor’s degree, with preferred skills including Databricks certifications and familiarity with ITIL service management processes"