Cloud Engineer Azure Databricks

VALCE T@lent Solutions; United States; Remoto; USD 150.000 - 210.000; Tempo pieno

Descrizione del lavoro

VALCE T@lent Solutions seeks an experienced cloud infrastructure engineer to design, implement, and manage the Azure data/AI platform. You will author Terraform modules for Databricks, ADLS Gen2, and Azure ML, configure secure hub-and-spoke networking, and optimize cost visibility with FinOps practices.

The role requires deep Azure expertise, Terraform mastery, and hands-on Databricks administration, including Unity Catalog, RBAC, and enterprise-scale ADLS Gen2 design.

Competenze

• 6+ years in cloud infrastructure or platform engineering.
• 4+ years focused on Microsoft Azure.
• Expert-level hands-on experience with Terraform.
• Databricks administration, including Unity Catalog and cluster policies.
• Deep knowledge of Azure Networking, Entra ID, and RBAC.
• Experience with ADLS Gen2 design at enterprise scale.
• Platform-side experience with Azure Data Factory.
• Proficiency in Python, PowerShell, and/or Bash.

Mansioni

• Design, implement, and administer the Azure foundation for the data and AI platform.
• Author and maintain reusable Terraform modules for Databricks, ADLS Gen2, and Azure ML.
• Implement private networking, hub-and-spoke topology, and security posture.
• Manage ADLS Gen2 storage account architecture and access patterns.
• Oversee Azure Data Factory integration runtime provisioning and deployment automation.
• Provision infrastructure for machine learning workloads including GPU quota and MLflow integration.
• Build monitoring layers using Azure Monitor and Log Analytics.
• Manage FinOps, cost visibility, and capacity planning for DBU and storage spend.
• Establish platform standards and reference configurations for Databricks administration.

Conoscenze

Cloud infrastructure

Microsoft Azure

Scripting (Python/Bash)

Terraform

Databricks administration

Unity Catalog

RBAC / Entra ID

Azure Networking

ADLS Gen2 design

Azure Data Factory

Azure Monitor / Log Analytics

Kubernetes / AKS

Policy as Code

Event Hubs / Kafka

Strumenti

Terraform

Databricks

Unity Catalog

Azure Data Factory

Azure Monitor

Log Analytics

ADLS Gen2

Azure Networking

Entra ID

RBAC

Kubernetes

AKS

MLflow

Databricks Asset Bundles

Descrizione del lavoro

Responsibilities

• Design, implement, and administer the Azure foundation for the data and AI platform
• Author and maintain reusable Terraform modules for Databricks, ADLS Gen2, and Azure ML
• Implement private networking, hub-and-spoke topology, and security posture
• Manage ADLS Gen2 storage account architecture and access patterns
• Oversee Azure Data Factory integration runtime provisioning and deployment automation
• Provision infrastructure for machine learning workloads including GPU quota and MLflow integration
• Build monitoring layers using Azure Monitor and Log Analytics
• Manage FinOps, cost visibility, and capacity planning for DBU and storage spend
• Establish platform standards and reference configurations for Databricks administration

Requirements

• 6+ years in cloud infrastructure or platform engineering
• 4+ years focused on Microsoft Azure
• Expert-level hands-on experience with Terraform
• Demonstrable experience in Databricks administration, including Unity Catalog and cluster policies
• Deep knowledge of Azure Networking, Entra ID, and RBAC
• Experience with ADLS Gen2 design at enterprise scale
• Platform-side experience with Azure Data Factory
• Proficiency in Python, PowerShell, and/or Bash

Preferred Qualifications

• Experience with Databricks Asset Bundles and workspace-as-code patterns
• Experience leading Unity Catalog migrations
• Knowledge of Azure Machine Learning and MLflow
• Experience with Kubernetes, AKS, and containerized workloads
• Familiarity with Policy as Code (Azure Policy, OPA, Sentinel)
• Experience with Event Hubs, Kafka, or Stream Analytics
• Formal FinOps practice experience