Databricks Platform Engineer

Lumen Resources; New York (NY); On-site; USD 160,000 - 210,000; Full time

Job summary

Lumen Resources is seeking an Azure Databricks Platform Engineer to own the administration, governance, security, and operational health of an enterprise Databricks environment.

This role focuses on platform health, Unity Catalog governance, access controls, workspaces, compute, production jobs, and cost-efficient operations across data, analytics, and AI workloads.

The position emphasizes platform administration over traditional data engineering, with automation and IaC experience beneficial.

Qualifications

• Hands-on Databricks administration in a production enterprise environment.
• Strong Unity Catalog governance, permissions, schemas, and access controls.
• Experience managing Databricks workspaces, clusters/compute, jobs, and platform configurations.

Responsibilities

• Administer Databricks environments across development to production workloads.
• Own day-to-day platform health: workloads, compute, availability, and operational activity.
• Administer Unity Catalog and scalable governance for data, permissions, and access.
• Manage users, groups, roles, and access controls with data protection.
• Troubleshoot production Databricks issues and platform connectivity.
• Configure and optimize clusters and compute resources for workloads and cost.
• Establish consistent admin and security standards across environments.
• Support controlled promotion of platform configurations and security standards.

Skills

Databricks administration

Unity Catalog

Workspace management

Cluster optimization

Security governance

Cost optimization

Azure cloud & cost sensing

Tools

Terraform

CI/CD

GitHub

Azure DevOps

Power BI

Microsoft Purview

Job description

We're looking for an Azure Databricks Platform Engineer to take ownership of the administration, governance, security, and operational health of an enterprise Databricks environment. This individual will ensure the platform remains secure, reliable, properly governed, and cost-efficient while supporting a growing portfolio of enterprise data, analytics, and AI workloads.

This is a platform administration role rather than a traditional data engineering position. The ideal candidate has hands‑on experience owning Databricks environments and understands how to manage Unity Catalog, access controls, workspaces, compute resources, production jobs, and platform governance.

What You'll Do

• Administer Databricks environments supporting development, testing, pre‑production, and production workloads.
• Own day‑to‑day platform health by monitoring workloads, scheduled jobs, compute resources, availability, and other operational activity.
• Administer Unity Catalog and maintain scalable governance standards for enterprise data, permissions, and access.
• Manage users, groups, roles, and access controls while ensuring sensitive information is appropriately protected across the platform.
• Troubleshoot Databricks platform issues involving permissions, data access, jobs, compute, workspace configuration, and connectivity.
• Configure and optimize clusters and compute resources based on workload requirements, performance, utilization, and cloud cost considerations.
• Establish consistent administrative and security standards as additional users, datasets, and business workloads are introduced to the platform.
• Support controlled promotion of platform configurations and security standards across development and production environments.

What We're Looking For

• Hands‑on experience administering Databricks in a production enterprise environment, with direct responsibility for the platform rather than solely developing notebooks or data pipelines.
• Strong experience with Unity Catalog, including governance, permissions, catalogs, schemas, users/groups, and enterprise data‑access controls.
• Experience administering Databricks workspaces, clusters/compute, jobs, security, permissions, and platform configurations.
• Demonstrated experience troubleshooting production Databricks issues and supporting the operational health of the platform.
• Understanding of role‑based access, least‑privilege security, sensitive‑data protection, and scalable enterprise governance practices.
• Experience monitoring and optimizing Databricks compute for performance, utilization, reliability, and cost.
• Experience supporting multiple Databricks environments and maintaining consistent configurations and security standards between them.
• Azure Databricks experience is strongly preferred; candidates with significant Databricks administration experience in another major cloud environment may also be considered.
• Familiarity with Azure identity, storage, networking, monitoring, or security services is preferred.
• Exposure to automation, Infrastructure as Code, Terraform, CI/CD, GitHub, or Azure DevOps is beneficial.
• Knowledge of Microsoft Purview, Microsoft Fabric, Delta Lake, Lakehouse architecture, Power BI, or enterprise AI integrations is a plus.

Join us at Lumen Resources, where your expertise will contribute to our growth, and you'll have the opportunity to have fun and thrive alongside a dynamic group of professionals.

Lumen Resources is an Equal Opportunity Employer. Don't hesitate to get in touch with our team for more information.