Sr. Software Engineer

Black Rock Groups; Maine; On-site; USD 120,000 - 180,000; Full time

Job summary

Black Rock Groups is seeking a BI Admin Databricks specialist to manage and optimize our Databricks workspace across development, testing, and production environments. The role emphasizes IAM, governance, and incident response within Azure Cloud.

You will configure cluster policies, manage entitlements, monitor performance, and contribute to CI/CD and IaC practices to ensure secure and scalable data pipelines.

Qualifications

• Experience with Azure Cloud and IAM controls.
• Experience with Databricks workspace admin and governance.
• Strong troubleshooting and cross-functional communication.
• Familiarity with monitoring, logging, and incident response.
• Automation scripting using Python, PowerShell, Bash, or Terraform.

Responsibilities

• Administer and support Databricks workspace across dev, test, and prod.
• Manage user provisioning, RBAC, service principals, groups, entitlements.
• Configure cluster policies, instance pools, job compute, and resources.
• Monitor health, performance, usage trends, and incidents.
• Implement secrets management, data access, encryption, and audit logging.
• Support Unity Catalog and data governance at workspace level.
• Develop automation for workspace setup, policy enforcement, monitoring.
• Support CI/CD and IaC for Databricks deployments.
• Troubleshoot platform, connectivity, identity federation, and job issues.
• Create operational docs, runbooks, support procedures.
• Understand Spark workflows, Delta Lake, and scheduling.

Skills

Azure Cloud

IAM & RBAC

Databricks admin

Networking concepts

Python/Automation

Monitoring & Alerts

CI/CD / IaC

Troubleshooting

Documentation

Collaboration

Tools

Terraform

PowerShell

Bash

Job description

BI Admin Databricks Job Description

Key Responsibilities

• Administer and support the Databricks workspace environment across development, test, and production.
• Manage user provisioning, role based access controls, service principles, groups, and entitlements.
• Configure and maintain cluster policies, instance pools, job compute, and all purpose compute resources.
• Monitor platform health, job performance, usage trends, and operational incidents.
• Implement security controls for secrets management, data access, encryption, private connectivity, and audit logging.
• Support Unity Catalog, data governance standards, and workspace level administration.
• Develop and maintain automation for workspace setup, policy enforcement, monitoring, and administrative tasks.
• Support CI/CD and infrastructure as code practices for Databricks deployments and configuration management.
• Troubleshoot platform, connectivity, identity federation, and job execution issues.
• Create and maintain operational documentation, runbooks, standards, and support procedures.
• Understanding data engineering workflows using Spark, notebooks, Delta Lake, and job scheduling.

Required Qualifications

• Experience with Azure Cloud.
• Strong understanding of identity and access management, including SSO, SCIM, RBAC, and service principals.
• Experience with Databricks cluster management, workspace administration, job orchestration, and governance controls.
• Knowledge of networking concepts such as VNet/VPC, private endpoints, firewalls, DNS, and secure connectivity.
• Experience with scripting or automation using Python, PowerShell, Bash, Terraform, or similar tools.
• Familiarity with monitoring, logging, alerting, and incident response processes.
• Strong troubleshooting, communication, and cross functional collaboration skills.

Key Skills

• Databricks Workspace Administration
• Unity Catalog and Governance
• Cluster Policies and Performance Tuning
• Cloud Platform Integration
• Security and Access Management
• Automation and Infrastructure as Code
• Cost Optimization
• Troubleshooting and Operational Support
• Documentation and Cross Functional Collaboration
• Logic Apps, APIM, App Gateways