Databricks Platform Engineer

Lumen Resources; Chicago (IL); On-site; USD 120,000 - 160,000; Full time

Job summary

Lumen Resources is seeking an Azure Databricks Platform Engineer to own the administration, governance, security, and operational health of our enterprise Databricks environment. This role emphasizes platform health, access controls, and cost-efficient operations across a growing data/analytics/AI portfolio.

The ideal candidate brings hands-on Databricks administration experience in production and a strong focus on Unity Catalog governance, RBAC, and multi-environment consistency.

Qualifications

• Hands-on Databricks administration in a production enterprise environment.
• Strong Unity Catalog governance and access controls.
• Experience managing multiple Databricks environments with consistent configurations and security standards.
• Familiarity with IaC, Terraform, CI/CD, GitHub, or Azure DevOps is beneficial.

Responsibilities

• Administer Databricks environments supporting development, testing, pre-production, and production workloads.
• Own day-to-day platform health by monitoring workloads, compute resources, and jobs.
• Administer Unity Catalog and maintain scalable governance standards for enterprise data and permissions.
• Manage users, groups, roles, and access controls across the platform.
• Troubleshoot production Databricks issues involving permissions, data access, jobs, and connectivity.
• Configure and optimize clusters and compute resources based on workload and cost considerations.
• Establish consistent administrative and security standards as new users and datasets are introduced.
• Support promotion of platform configurations and security standards across environments.

Skills

Databricks administration

Unity Catalog governance

RBAC & access controls

Platform monitoring & troubleshooting

Cost optimization

Tools

Terraform

GitHub

Azure DevOps

Job description

Azure Databricks Platform Engineer to take ownership of the administration, governance, security, and operational health of an enterprise Databricks environment. This individual will ensure the platform remains secure, reliable, properly governed, and cost-efficient while supporting a growing portfolio of enterprise data, analytics, and AI workloads.

This is a platform administration role rather than a traditional data engineering position. The ideal candidate has hands-on experience owning Databricks environments and understands how to manage Unity Catalog, access controls, workspaces, compute resources, production jobs, and platform governance.

What You'll Do

• Administer Databricks environments supporting development, testing, pre-production, and production workloads.
• Own day-to-day platform health by monitoring workloads, scheduled jobs, compute resources, availability, and other operational activity.
• Administer Unity Catalog and maintain scalable governance standards for enterprise data, permissions, and access.
• Manage users, groups, roles, and access controls while ensuring sensitive information is appropriately protected across the platform.
• Troubleshoot Databricks platform issues involving permissions, data access, jobs, compute, workspace configuration, and connectivity.
• Configure and optimize clusters and compute resources based on workload requirements, performance, utilization, and cloud cost considerations.
• Establish consistent administrative and security standards as additional users, datasets, and business workloads are introduced to the platform.
• Support controlled promotion of platform configurations and security standards across development and production environments.

What We're Looking For

• Hands-on experience administering Databricks in a production enterprise environment, with direct responsibility for the platform rather than solely developing notebooks or data pipelines.
• Strong experience with Unity Catalog, including governance, permissions, catalogs, schemas, users/groups, and enterprise data-access controls.
• Experience administering Databricks workspaces, clusters/compute, jobs, security, permissions, and platform configurations.
• Demonstrated experience troubleshooting production Databricks issues and supporting the operational health of the platform.
• Understanding of role-based access, least-privilege security, sensitive-data protection, and scalable enterprise governance practices.
• Experience monitoring and optimizing Databricks compute for performance, utilization, reliability, and cost.Least-privilege security, sensitive-data protection, and scalable enterprise governance practices.Experience monitoring and optimizing Databricks compute for performance, utilization, reliability, and cost.Experience monitoring and optimizing Databricks compute for performance, utilization, reliability, and cost.Experience monitoring and optimizing Databricks compute for performance, utilization, reliability, and cost.Executioning. (end of line)Least-privilege security, sensitive-data protection, and scalable enterprise governance practices.Experience monitoring and optimizing Databricks compute for performance, utilization, reliability, and cost.Experience monitoring and optimizing Databricks compute for performance, utilization, reliability, and cost.Experience monitoring and optimizing Databricks compute for performance, utilization, reliability, and cost.Experience monitoring and optimizing Databricks compute for performance, utilization, reliability, and cost.Experience monitoring and optimizing Databricks compute for performance, utilization, reliability, and cost.
• Experience supporting multiple Databricks environments and maintaining consistent configurations and security standards between them.
• Azure Databricks experience is strongly preferred; candidates with significant Databricks administration experience in another major cloud environment may also be considered.
• Familiarity with Azure identity, storage, networking, monitoring, or security services is preferred.
• Exposure to automation, Infrastructure as Code, Terraform, CI/CD, GitHub, or Azure DevOps is beneficial.
• Knowledge of Microsoft Purview, Microsoft Fabric, Delta Lake, Lakehouse architecture, Power BI, or enterprise AI integrations is a plus.

Join us at Lumen Resources, where your expertise will contribute to our growth, and you'll have the opportunity to have fun and thrive alongside a dynamic group of professionals.

Lumen Resources:

At Lumen Resources you will be partnering with trained recruiters who will guide you through discovering a brighter future. Our recruiters are concise and efficient in all interactions, hold exceptional communication, and will support resume writing and interview preparation, so you can land any position you want!

Lumen Resources is an Equal Opportunity Employer.