Databricks Platform Engineer

EXL; New York (NY); On-site; USD 140,000 - 200,000; Full time

Job summary

EXL is seeking a Databricks Platform Manager/Engineer to design, administer, secure, and optimize our enterprise Databricks platform on AWS. You will own the platform lifecycle, ensuring scalability, governance, cost optimization, and operational excellence to enable data engineering, analytics, and AI teams.

This role focuses on platform engineering rather than data engineering or analytics development, with responsibilities spanning workspace design, security, RBAC, CI/CD, and monitoring.

Qualifications

• 5–10 years IT experience with 3–5+ years in Databricks on AWS.
• Hands-on administration of Databricks platforms in production.
• Experience designing, securing, and optimizing enterprise Databricks platforms.

Responsibilities

• Design, deploy, and manage Databricks workspaces across environments.
• Establish workspace standards, networking, and isolation.
• Manage DBR version planning, testing, rollout.
• Maintain availability, reliability, and performance.
• Create and manage cluster policies and resource configurations.
• Integrate Databricks with AWS IAM and manage authentication.
• Administer Unity Catalog and RBAC.
• Enable CI/CD pipelines and IaC using Terraform.
• Monitor workspace health, incidents, and dashboards.
• Cost optimization and governance of platform usage.

Skills

Databricks Platform

Workspace administration

Cluster management

Unity Catalog

AWS

IAM

Terraform

DevOps

CI/CD

Git

Monitoring

Security

Platform monitoring

Education

Bachelor's degree in Computer Science or related field

Tools

Terraform

Databricks Runtime

Unity Catalog

GitHub Actions

Jenkins

Azure DevOps

CloudWatch

Job description

Experience

5–10 years of overall IT experience, including 3–5+ years of hands‑on experience administering and managing Databricks platforms in production environments on AWS.

Job Summary

We are looking for a Databricks Platform Manager/Engineer responsible for designing, administering, securing, and optimizing the enterprise Databricks platform on AWS. The ideal candidate will own the platform lifecycle, ensuring scalability, security, governance, cost optimization, and operational excellence while enabling data engineering, analytics, and AI teams to work efficiently.

This role focuses on platform engineering rather than data engineering or analytics development.

Key Responsibilities

--Platform Administration

• Design, deploy, and manage Databricks workspaces across Development, Test, UAT, and Production environments.
• Establish workspace standards, networking, and environment isolation.
• Manage platform upgrades, including Databricks Runtime (DBR) version planning, testing, and rollout.
• Maintain platform availability, reliability, and performance.
• Create and manage cluster policies to enforce organizational standards.
• Configure instance pools and job clusters.
• Prevent oversized or non‑compliant clusters.
• Standardize cluster configurations across teams.
• Optimize cluster startup times and resource utilization.

--Security & Identity Management

• Integrate Databricks with AWS IAM.
• Configure and manage authentication and authorization.
• Manage service principals, personal access tokens (PATs), and secret scopes.
• Implement least‑privilege access controls.
• Support enterprise identity management and SSO.

--Governance & Data Access

• Administer Unity Catalog.
• Manage catalogs, schemas, tables, and permissions.
• Implement role‑based access control (RBAC).
• Enforce data governance and access policies.
• Support audit and compliance requirements.

--DevOps & CI/CD

• Configure Git integration with Databricks Repos.
• Support CI/CD pipelines for notebooks, workflows, and infrastructure.
• Collaborate with DevOps teams on automated deployments.
• Enable Infrastructure as Code (IaC) using Terraform where applicable.

--Monitoring & Operations

• Monitor workspace health, jobs, clusters, and platform performance.
• Respond to incidents and perform root cause analysis.
• Maintain operational dashboards and alerts.
• Support production releases and platform maintenance activities.

--Cost Optimization

• Monitor Databricks usage and cloud spending.
• Recommend cluster sizing and auto‑scaling strategies.
• Optimize job scheduling and compute utilization.
• Track cost trends and recommend savings opportunities.

--Compliance & Audit

• Configure audit logging.
• Ensure platform complies with enterprise security policies.
• Support internal and external audits.
• Maintain operational documentation and platform standards.

Required Skills

• Databricks Platform
• Workspace administration
• Cluster management
• Cluster policies
• Instance pools
• Databricks Runtime management
• Workflows
• Databricks Repos
• Unity Catalog
• Personal Access Tokens (PATs)
• AWS
• IAM
• EC2 concepts
• S3
• CloudWatch
• KMS
• Security Groups
• DevOps
• Git
• CI/CD
• Terraform
• Infrastructure as Code
• Azure DevOps, GitHub Actions, or Jenkins
• Monitoring
• Platform monitoring
• Performance tuning
• Required Experience

Candidates should have hands‑on experience with:

• Managing enterprise Databricks platforms
• Setting up multiple Databricks workspaces
• Configuring Unity Catalog
• AWS IAM integration
• Cluster policy creation and governance
• CI/CD implementation
• Git integration
• Platform monitoring and troubleshooting
• Secrets management
• Audit logging
• Production support

Preferred Qualifications

• Experience supporting multiple business units or enterprise data platforms.
• Hands‑on experience with Terraform for Databricks infrastructure provisioning.
• Knowledge of Lakehouse architecture and Delta Lake.
• Experience with FinOps practices for cloud cost optimization.
• Familiarity with data governance and security frameworks.

Nice to Have

• Experience with AWS Organizations and Control Tower.
• Knowledge of networking concepts for private connectivity (PrivateLink, VPC endpoints).
• Experience with monitoring tools such as CloudWatch, Datadog, or Splunk.
• Exposure to Apache Spark internals.
• Experience supporting AI/ML workloads on Databricks.
• Education
• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
• Certifications (Preferred)
• Databricks Certified Data Engineer Professional or Databricks Certified Platform Administrator (if available).
• AWS Certified Solutions Architect – Associate/Professional.
• AWS Certified SysOps Administrator.
• Terraform Associate certification.

Ideal Candidate Profile

The ideal candidate has a strong platform engineering and cloud infrastructure background with deep expertise in Databricks administration on AWS. They are comfortable owning the operational health, security, governance, automation, and cost optimization of the Databricks platform while partnering closely with data engineering, analytics, DevOps, and security teams. Experience with Terraform is highly desirable, as many enterprise Databricks environments are provisioned and managed through Infrastructure as Code.