Databricks Platform Engineer

EXL Talent Acquisition Team

New York (NY)

On-site

USD 180,000 - 240,000

Full time

2 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

ExlService Holdings, Inc. seeks a Databricks Platform Manager/Engineer (AWS) to own the enterprise Databricks platform lifecycle, ensuring scalability, security, governance, cost optimization, and operational excellence across Development, Test, UAT and Production environments.

This role focuses on platform engineering rather than data engineering, enabling data teams to work efficiently while maintaining strong platform standards and incident response capabilities.

Qualifications

  • 8–10 years IT experience with 5+ years on Databricks on AWS.
  • Design, secure, and optimize enterprise Databricks platform.
  • Unity Catalog, RBAC, IAM integration and governance experience.
  • Hands-on Terraform and CI/CD in production environments.

Responsibilities

  • Platform Administration: design, deploy, and manage Databricks workspaces across environments.
  • Compute & Cluster Management: enforce policies, configure pools, optimize startup times.
  • Security & Identity: implement least-privilege access, SSO, PATs, secret scopes.
  • Governance & Data Access: administer Unity Catalog and RBAC, enforce policies.
  • DevOps & CI/CD: integrate Databricks with repos, support notebooks and infra deployments.
  • Monitoring & Operations: monitor health, respond to incidents, maintain dashboards.
  • Cost Optimization: monitor usage and optimize cluster sizing and autoscaling.
  • Compliance & Audit: enable audit logging and support internal/external audits.

Skills

Databricks Platform
Workspace administration
Cluster management
Unity Catalog
IAM integration
DevOps
Terraform
CI/CD
AWS
Cloud networking
Monitoring
Security

Education

Bachelor's degree in Computer Science/Engineering/IT
Databricks Certifications (Preferred)
AWS Certifications (Optional)

Tools

Terraform
CI/CD
Databricks Repos
Git integration
IAM

Job description

Databricks Platform Manager / Engineer (AWS)
Experience

8–10 years of overall IT experience, including 5+ years of hands‑on experience administering and managing Databricks platforms in production environments on AWS.

Job Summary

We are looking for a Databricks Platform Manager/Engineer responsible for designing, administering, securing, and optimizing the enterprise Databricks platform on AWS. The ideal candidate will own the platform lifecycle, ensuring scalability, security, governance, cost optimization, and operational excellence while enabling data engineering, analytics, and AI teams to work efficiently.

This role focuses on platform engineering rather than data engineering or analytics development.

Key Responsibilities
Platform Administration
  • Strong hands‑on experience in System Design and Architecture, including making architecture decisions.
  • Strong AWS and DevOps experience.
  • Hands‑on experience with Terraform and CI/CD pipelines.
  • Strong understanding of Cloud Networking, Security, Scalability, and High Availability (HA).
  • Good production troubleshooting and problem‑solving skills.
  • Design, deploy, and manage Databricks workspaces across Development, Test, UAT, and Production environments.
  • Establish workspace standards, networking, and environment isolation.
  • Manage platform upgrades, including Databricks Runtime (DBR) version planning, testing, and rollout.
  • Maintain platform availability, reliability, and performance.
  • Compute & Cluster Management
  • Create and manage cluster policies to enforce organizational standards.
  • Configure instance pools and job clusters.
  • Prevent oversized or non‑compliant clusters.
  • Standardize cluster configurations across teams.
  • Optimize cluster startup times and resource utilization.
  • Security & Identity Management
  • Integrate Databricks with AWS IAM.
  • Configure and manage authentication and authorization.
  • Manage service principles, personal access tokens (PATs), and secret scopes.
  • Implement least‑privilege access controls.
  • Support enterprise identity management and SSO.
  • Governance & Data Access
  • Administer Unity Catalog.
  • Manage catalogs, schemas, tables, and permissions.
  • Implement role‑based access control (RBAC).
  • Enforce data governance and access policies.
  • Support audit and compliance requirements.
  • DevOps & CI/CD
  • Configure Git integration with Databricks Repos.
  • Support CI/CD pipelines for notebooks, workflows, and infrastructure.
  • Collaborate with DevOps teams on automated deployments.
  • Enable Infrastructure as Code (IaC) using Terraform where applicable.
Monitoring & Operations
  • Monitor workspace health, jobs, clusters, and platform performance.
  • Respond to incidents and perform root cause analysis.
  • Maintain operational dashboards and alerts.
  • Support production releases and platform maintenance activities.
  • Cost Optimization
  • Monitor Databricks usage and cloud spending.
  • Recommend cluster sizing and auto‑scaling strategies.
  • Optimize job scheduling and compute utilization.
  • Track cost trends and recommend savings opportunities.
  • Compliance & Audit
  • Configure audit logging.
  • Ensure platform complies with enterprise security policies.
  • Support internal and external audits.
  • Maintain operational documentation and platform standards
Required Skills
  • Databricks Platform
  • Workspace administration
  • Cluster management
  • Cluster policies
  • Instance pools
  • Job clusters
  • Databricks Runtime management
  • Workflows
  • Databricks Repos
  • Unity Catalog
  • Secret Scopes
  • Service Principals
  • Personal Access Tokens (PATs)
  • AWS
  • IAM
  • VPC fundamentals
  • EC2 concepts
  • S3
  • CloudWatch
  • KMS
  • Security Groups
  • Networking fundamentals
  • DevOps
  • Git
  • CI/CD
  • Terraform
  • Infrastructure as Code
  • Azure DevOps, GitHub Actions, or Jenkins
  • Monitoring
  • Platform monitoring
  • Incident management
  • Log analysis
  • Performance tuning
  • Cost optimization
  • Required Experience
Candidates should have hands‑on experience with:
  • Managing enterprise Databricks platforms
  • Setting up multiple Databricks workspaces
  • Configuring Unity Catalog
  • AWS IAM integration
  • Cluster policy creation and governance
  • Runtime version upgrades
  • CI/CD implementation
  • Git integration
  • Platform monitoring and troubleshooting
  • Secrets management
  • Audit logging
  • Good Cloud, DevOps, and system Design Experience.
  • Hands‑on terraform and CI/CD.
  • Strong understanding of cloud networking, security , scalability, and HA
  • Good production troubleshooting skills.
  • Production support
  • Cost optimization initiatives
  • Preferred Qualifications
  • Experience managing large‑scale Databricks environments (100+ users).
  • Experience supporting multiple business units or enterprise data platforms.
  • Hands‑on experience with Terraform for Databricks infrastructure provisioning.
  • Knowledge of Lakehouse architecture and Delta Lake.
  • Experience with FinOps practices for cloud cost optimization.
  • Familiarity with data governance and security frameworks.
Nice to Have
  • Experience with AWS Organizations and Control Tower
  • Knowledge of networking concepts for private connectivity (PrivateLink, VPC endpoints).
  • Experience with monitoring tools such as CloudWatch, Datadog, or Splunk
  • Exposure to Apache Spark internals
  • Experience supporting AI/ML workloads on Databricks
Education
  • Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field
  • Certifications (Preferred)
  • Databricks Certified Data Engineer Professional or Databricks Certified Platform Administrator (if available)
  • AWS Certified Solutions Architect – Associate/Professional
  • AWS Certified SysOps Administrator
  • Terraform Associate certification
  • Ideal Candidate Profile

The ideal candidate has a strong platform engineering and cloud infrastructure background with deep expertise in Databricks administration on AWS. They are comfortable owning the operational health, security, governance, automation, and cost optimization of the Databricks platform while partnering closely with data engineering, analytics, DevOps, and security teams. Experience with Terraform is highly desirable, as many enterprise Databricks environments are provisioned and managed through Infrastructure as Code.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Databricks Administrator / Platform Engineer
Databricks Administrator / Platform Engineer

Changeis • United States

Remote
USD 130,000 - 190,000
Cloud Engineer Azure Databricks
Cloud Engineer Azure Databricks

Turtle Trax S.A. • United States

Remote
USD 140,000 - 190,000
Databricks Data Architect
Databricks Data Architect

NewVision • San Francisco (CA)

Hybrid
USD 140,000 - 190,000
Hybrid work model
AWS Databricks Platform Administrator
AWS Databricks Platform Administrator

Veriipro • Raritan (NJ)

On-site
USD 150,000 - 210,000
Databricks Data Platform Architect
Databricks Data Platform Architect

UsefulBI Corporation • Raleigh (NC)

On-site
USD 130,000 - 180,000
Senior Data Architect – Databricks
Senior Data Architect – Databricks

MatrixEdge Inc • Kansas City (MO)

On-site
USD 150,000 - 190,000
Databricks Data Architect
Databricks Data Architect

NewVision Software • United States

Remote
USD 120,000 - 180,000
Sr. Staff Platform/Data Reliability Engineer, Databricks (R5537)
Sr. Staff Platform/Data Reliability Engineer, Databricks (R5537)

Remote • San Francisco (CA)

On-site
USD 200,000 - 270,000
Databricks Data Engineer
Databricks Data Engineer

EXL Talent Acquisition Team • New York (NY)

On-site
USD 170,000 - 230,000
Databricks Platform Administrator – Azure & AWS
Databricks Platform Administrator – Azure & AWS

Vibhutechinc • Miami (FL)

On-site
USD 120,000 - 170,000