Databricks Platform Engineer

Tata Consultancy Services; Quincy (MA); On-site; USD 100,000 - 117,000; Full time

Benefits offered by this job

Medical coverage

401K plan

Performance bonus

Training reimbursement

Vacation & holidays

Job summary

Tata Consultancy Services seeks an experienced platform/infra engineer to design, deploy, and manage Databricks workloads across Azure subscriptions, with IaC using Terraform and strong security controls.

You will build CI/CD pipelines with Azure DevOps, optimize costs, mentor teammates, and collaborate with data engineers to deliver scalable data platforms in a compliant, well-documented environment.

Qualifications

• 7+ years of platform or infra engineering, or DevOps experience.
• 2+ years hands-on Databricks infrastructure setup.
• 3+ years with Azure cloud services.
• 2+ years with Azure DevOps (Pipelines, Repos, Artifacts).
• Strong Infrastructure-as-Code experience with Terraform.
• Solid CI/CD knowledge and implementation.
• Experience with Linux/Unix and scripting (Bash, Python).
• Familiarity with Docker, Kubernetes, networking, security, and IAM in cloud environments.

Responsibilities

• Design, deploy, and manage Databricks workspaces across environments.
• Implement IaC solutions using Terraform for Databricks resources.
• Establish security best practices, network isolation, and data governance.
• Configure cluster management, auto-scaling, and cost optimization.
• Build CI/CD pipelines with Azure DevOps for notebooks and configs.
• Develop runbooks and governance documentation.
• Mentor team members on Databricks and Azure DevOps practices.

Skills

Databricks

Azure

Azure DevOps

Terraform

CI/CD

Linux/Unix

Docker

Kubernetes

Security & IAM

Networking

Education

Bachelor's degree in Computer Science

Tools

Terraform Enterprise

Azure DevOps

Databricks

CI/CD tooling

Job description

Job Description

Must Have Technical/Functional Skills

Platform & Infrastructure

• Design, deploy, and manage Databricks workspaces across multiple Azure subscriptions and environments (dev, staging, production)
• Implement Infrastructure-as-Code (IaC) solutions using Terraform for Databricks resources
• Establish and maintain security best practices, including network isolation, RBAC, and data governance
• Configure and optimize cluster management, including auto-scaling policies and cost optimization strategies
• Manage Databricks workspace connectivity and networking (VNets, private endpoints, firewalls)

DevOps & Automation

• Build CI/CD pipelines using Azure DevOps for deploying Databricks notebooks, jobs, and configurations
• Implement automated testing and validation of data pipelines
• Create self-service deployment frameworks and templates for development teams
• Establish monitoring, alerting, and logging solutions for Databricks environments
• Automate workspace provisioning and deprovisioning processes

Collaboration & Enablement

• Partner with data engineers and analysts to understand platform requirements
• Provide technical guidance and documentation for platform usage
• Develop runbooks and operational procedures for incident response
• Mentor team members on Databricks best practices and Azure DevOps tools

Governance & Compliance

• Implement access controls and audit logging across Databricks workspaces
• Ensure compliance with organizational security and data governance policies
• Monitor and manage workspace usage, costs, and resource utilization
• Document architecture decisions and maintain platform documentation

Roles & Responsibilities

Required Qualifications

• 7+ years of experience in platform engineering, infrastructure engineering, or DevOps
• 2+ years of hands‑on experience with Databricks infrastructure setup (or Apache Spark/Delta Lake environments)
• 3+ years of experience with Azure cloud services and infrastructure
• 2+ years of experience with Azure DevOps (Pipelines, Repos, Artifacts)
• Strong proficiency in Infrastructure-as-Code tools (Terraform Enterprise)
• Solid understanding of CI/CD principles and implementation
• Experience with Linux/Unix environments and scripting (Bash, Python, or similar)
• Familiarity with containerization technologies (Docker, Kubernetes) Strong knowledge of networking, security, and identity management in cloud environments

TCS Employee Benefits Summary

• Discretionary Annual Incentive.
• Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
• Family Support: Maternal & Parental Leaves.
• Insurance Options: Auto & Home Insurance, Identity Theft Protection.
• Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.
• Time Off: Vacation, Time Off, Sick Leave & Holidays.
• Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.

Salary Range: $100,000 - $117,000 a year

Qualifications: BACHELOR OF COMPUTER SCIENCE