Databricks Platform Engineer

Tata Consultancy Services; Quincy (MA); On-site; USD 100,000 - 117,000; Full time

Benefits offered by this job

Discretionary Annual Incentive

Comprehensive Medical Coverage: Health

401K Plan

Certification & Training Reimbursement

Commuter Benefits

Job summary

Tata Consultancy Services Limited in Quincy, MA seeks an experienced Platform Engineer to design, deploy, and manage Databricks infrastructure on Azure, implementing IaC with Terraform, and enforcing security and cost controls.

You will partner with data engineers to create runbooks, automate provisioning, and ensure governance across Databricks environments using Azure DevOps for CI/CD, monitoring, and incident response.

Qualifications

• 7+ years of experience in platform engineering, infrastructure engineering, or DevOps.
• 2+ years of hands-on experience with Databricks infrastructure setup (or Apache Spark/Delta Lake environments).
• 3+ years of experience with Azure cloud services and infrastructure.
• 2+ years of experience with Azure DevOps (Pipelines, Repos, Artifacts).
• Strong proficiency in Infrastructure-as-Code tools (Terraform Enterprise).
• Solid understanding of CI/CD principles and implementation.
• Experience with Linux/Unix environments and scripting (Bash, Python, or similar).
• Familiarity with containerization technologies (Docker, Kubernetes).
• Strong knowledge of networking, security, and identity management in cloud environments.

Responsibilities

• Design, deploy, and manage Databricks workspaces across Azure subscriptions and environments.
• Implement IaC solutions using Terraform for Databricks resources.
• Establish and maintain security best practices, including RBAC and data governance.
• Configure and optimize Databricks cluster management and cost strategies.
• Manage workspace connectivity and networking (VNets, private endpoints, firewalls).
• Build CI/CD pipelines using Azure DevOps for Databricks notebooks, jobs, and configurations.
• Implement automated testing and validation of data pipelines.
• Create self-service deployment frameworks and templates for development teams.
• Establish monitoring, alerting, and logging solutions for Databricks environments.
• Automate workspace provisioning and deprovisioning processes.
• Partner with data engineers and analysts to understand platform requirements.
• Provide technical guidance and documentation for platform usage.
• Develop runbooks and operational procedures for incident response.
• Mentor team members on Databricks best practices and Azure DevOps tools.

Skills

Azure

DevOps

Linux/Unix

Scripting (Bash/Python)

Networking & security in cloud

Education

Bachelor of Computer Science

Tools

Terraform Enterprise

Docker

Kubernetes

Azure DevOps

Job description

Must Have Technical/Functional Skills

Key Responsibilities

Platform & Infrastructure

• Design, deploy, and manage Databricks workspaces across multiple Azure subscriptions and environments (dev, staging, production)
• Implement Infrastructure-as-Code (IaC) solutions using Terraform for Databricks resources
• Establish and maintain security best practices, including network isolation, RBAC, and data governance
• Configure and optimize cluster management, including auto-scaling policies and cost optimization strategies
• Manage Databricks workspace connectivity and networking (VNets, private endpoints, firewalls)

DevOps & Automation

• Build CI/CD pipelines using Azure DevOps for deploying Databricks notebooks, jobs, and configurations
• Implement automated testing and validation of data pipelines
• Create self-service deployment frameworks and templates for development teams
• Establish monitoring, alerting, and logging solutions for Databricks environments
• Automate workspace provisioning and deprovisioning processes

Collaboration & Enablement

• Partner with data engineers and analysts to understand platform requirements
• Provide technical guidance and documentation for platform usage
• Develop runbooks and operational procedures for incident response
• Mentor team members on Databricks best practices and Azure DevOps tools

Governance & Compliance

• Implement access controls and audit logging across Databricks workspaces
• Ensure compliance with organizational security and data governance policies
• Monitor and manage workspace usage, costs, and resource utilization
• Document architecture decisions and maintain platform documentation

Roles & Responsibilities

Required Qualifications

• 7+ years of experience in platform engineering, infrastructure engineering, or DevOps
• 2+ years of hands‑on experience with Databricks infrastructure setup (or Apache Spark/Delta Lake environments)
• 3+ years of experience with Azure cloud services and infrastructure
• 2+ years of experience with Azure DevOps (Pipelines, Repos, Artifacts)
• Strong proficiency in Infrastructure-as-Code tools (Terraform Enterprise)
• Solid understanding of CI/CD principles and implementation
• Experience with Linux/Unix environments and scripting (Bash, Python, or similar)
• Familiarity with containerization technologies (Docker, Kubernetes)
• Strong knowledge of networking, security, and identity management in cloud environments

TCS Employee Benefits Summary:

• Discretionary Annual Incentive.
• Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
• Family Support: Maternal & Parental Leaves.
• Insurance Options: Auto & Home Insurance, Identity Theft Protection.
• Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.
• Time Off: Vacation, Time Off, Sick Lea ve & Holidays.
• Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.
• Salary Range: $100,000-$117,000 a year

Location

Quincy, MA

Function

TECHNOLOGY

Role

Engineer

ID

435521

Skills

Azure | DevOps

Salary Range

$100,000 - $117,000

Desired Candidate Profile

Qualifications

BACHELOR OF COMPUTER SCIENCE