Databricks Platform Engineer - Azure, IaC & DevOps Expert

Tata Consultancy Services Limited; Quincy (MA); On-site; USD 100,000 - 117,000; Full time

Benefits offered by this job

Discretionary Annual Incentive

Comprehensive Medical Coverage: Health

401K Plan

Certification & Training Reimbursement

Commuter Benefits

Job summary

Tata Consultancy Services Limited in Quincy, MA seeks an experienced Platform Engineer to design, deploy, and manage Databricks infrastructure on Azure, implementing IaC with Terraform, and enforcing security and cost controls.

You will partner with data engineers to create runbooks, automate provisioning, and ensure governance across Databricks environments using Azure DevOps for CI/CD, monitoring, and incident response.

Qualifications

• 7+ years of experience in platform engineering, infrastructure engineering, or DevOps.
• 2+ years of hands-on experience with Databricks infrastructure setup (or Apache Spark/Delta Lake environments).
• 3+ years of experience with Azure cloud services and infrastructure.
• 2+ years of experience with Azure DevOps (Pipelines, Repos, Artifacts).
• Strong proficiency in Infrastructure-as-Code tools (Terraform Enterprise).
• Solid understanding of CI/CD principles and implementation.
• Experience with Linux/Unix environments and scripting (Bash, Python, or similar).
• Familiarity with containerization technologies (Docker, Kubernetes).
• Strong knowledge of networking, security, and identity management in cloud environments.

Responsibilities

• Design, deploy, and manage Databricks workspaces across Azure subscriptions and environments.
• Implement IaC solutions using Terraform for Databricks resources.
• Establish and maintain security best practices, including RBAC and data governance.
• Configure and optimize Databricks cluster management and cost strategies.
• Manage workspace connectivity and networking (VNets, private endpoints, firewalls).
• Build CI/CD pipelines using Azure DevOps for Databricks notebooks, jobs, and configurations.
• Implement automated testing and validation of data pipelines.
• Create self-service deployment frameworks and templates for development teams.
• Establish monitoring, alerting, and logging solutions for Databricks environments.
• Automate workspace provisioning and deprovisioning processes.
• Partner with data engineers and analysts to understand platform requirements.
• Provide technical guidance and documentation for platform usage.
• Develop runbooks and operational procedures for incident response.
• Mentor team members on Databricks best practices and Azure DevOps tools.

Skills

Azure

DevOps

Linux/Unix

Scripting (Bash/Python)

Networking & security in cloud

Education

Bachelor of Computer Science

Tools

Terraform Enterprise

Docker

Kubernetes

Azure DevOps

Job description

Tata Consultancy Services Limited in Quincy, MA seeks an experienced Platform Engineer to design, deploy, and manage Databricks infrastructure on Azure, implementing IaC with Terraform, and enforcing security and cost controls.

You will partner with data engineers to create runbooks, automate provisioning, and ensure governance across Databricks environments using Azure DevOps for CI/CD, monitoring, and incident response.