Sr. Azure Databricks Admin (Onsite)

Cognizant; Buffalo (NY); On-site; USD 140,000 - 170,000; Full time

Job summary

Cognizant in Buffalo, NY seeks a skilled Azure Databricks engineer to administer Databricks workspaces, clusters, and Unity Catalog across development, staging, and production environments. You will design and enforce cluster policies and access controls while integrating Databricks with ADLS Gen2, Key Vault, Active Directory, Event Hubs, and Synapse Analytics.

Monitor health, optimize costs, and drive CI/CD pipelines using Azure DevOps or GitHub Actions.

Responsibilities

• Administer and maintain Azure Databricks workspaces, clusters, and Unity Catalog across dev, staging, and production environments.
• Design and enforce cluster policies, access controls, and workspace configurations aligned with enterprise security standards.
• Manage Unity Catalog — including metastores, catalogs, schemas, and fine-grained data access policies.
• Monitor platform health, cluster utilization, job performance, and implement auto-scaling and cost optimization strategies.
• Integrate Databricks with Azure services including ADLS Gen2, Azure Key Vault, Azure Active Directory, Event Hubs, and Synapse Analytics.
• Implement secrets management, service principals, and managed identity configurations using Azure AD and Databricks Secrets.
• Automate infrastructure provisioning using Terraform, ARM/Bicep templates, or Databricks Terraform provider.
• Establish and maintain CI/CD pipelines for Databricks workflows using Azure DevOps or GitHub Actions.
• Collaborate with security teams to ensure compliance with data governance, audit logging, and lineage tracking.
• Troubleshoot cluster failures, job bottlenecks, Spark performance issues, and data pipeline errors.
• Maintain runbooks, platform documentation, and administer change management processes.

Job description

Please note this role is not able to offer visa transfer or sponsorship now or in the future

• Administer and maintain Azure Databricks workspaces, clusters, and Unity Catalog across dev, staging, and production environments.
• Design and enforce cluster policies, access controls, and workspace configurations aligned with enterprise security standards.
• Manage Unity Catalog — including metastores, catalogs, schemas, and fine-grained data access policies.
• Monitor platform health, cluster utilization, job performance, and implement auto-scaling and cost optimization strategies.
• Integrate Databricks with Azure services including ADLS Gen2, Azure Key Vault, Azure Active Directory, Event Hubs, and Synapse Analytics.
• Implement secrets management, service principals, and managed identity configurations using Azure AD and Databricks Secrets.
• Automate infrastructure provisioning using Terraform, ARM/Bicep templates, or Databricks Terraform provider.
• Establish and maintain CI/CD pipelines for Databricks workflows using Azure DevOps or GitHub Actions.
• Collaborate with security teams to ensure compliance with data governance, audit logging, and lineage tracking.
• Troubleshoot cluster failures, job bottlenecks, Spark performance issues, and data pipeline errors.
• Maintain runbooks, platform documentation, and administer change management processes.