Databricks Platform Expert

MICHELIN France; Pune District; On-site; INR 1,800,000 - 3,200,000; Full time

Job summary

MICHELIN France seeks a Databricks Platform Expert to manage and optimize our Databricks environment. You will handle workspaces, clusters, and notebooks while implementing Unity Catalog, RBAC, and secure access.

The role covers production-grade ETL pipelines with Delta Lake and Delta Live Tables, ensuring secure data governance and cost efficiency. You will collaborate with data engineering teams to debug Spark jobs, tune performance, and enforce CI/CD pipelines using Git.

Qualifications

• 4–6 years of experience with Databricks as an administrator or data engineer.
• Strong expertise in Apache Spark programming (PySpark preferred; SQL or Scala is a plus).
• Hands-on experience with Databricks Jobs, cluster configuration, SQL Warehouses, and Unity Catalog.
• Deep understanding of Delta Lake, ACID transactions, and lakehouse architecture.
• Experience with Git, CI/CD, and DevOps concepts for data engineering workflows.
• Knowledge of cloud platforms (Azure).
• Familiarity with IAM, networking basics, monitoring tools, and security patterns.

Responsibilities

• Manage, configure, and administer Databricks workspaces, clusters, SQL warehouses, serverless, jobs, and workspace objects.
• Implement and manage Unity Catalog, including catalogs, schemas, tables, access controls, and data lineage.
• Optimize cluster policies, auto-scaling strategies, and cost management for Serverless and Classic compute.
• Serve as the SME for Databricks infrastructure, governance, and security best practices.
• Monitor workspace performance, cluster stability, logs, job reliability, and platform health.
• Implement CI/CD pipelines for notebooks, jobs, and Delta Live Tables using Git integration.
• Support user provisioning, access controls (ACLs), secrets management, and workspace SSO.
• Write efficient Spark (PySpark / SQL / Scala) code for ETL, data transformations, and pipeline optimizations.
• Assist data engineering teams with Spark job debugging, performance tuning, and code reviews.
• Build and maintain production-grade pipelines leveraging Delta Lake, Databricks Jobs, and DLT.
• Implement and manage RBAC, SCIM provisioning, AIM, service principals, and cluster access controls.
• Ensure compliance with enterprise data governance, audit, and logging requirements.
• Manage secrets Key Vault and enforce secure credential handling.
• Support audit reports, compliance reviews, and workspace security configuration.
• Monitor job failures, cluster lifecycle performance, and system events using Databricks logs and cloud-native monitoring tools (Azure Monitor).
• Create automated alerts and observability dashboards for platform usage, cost, and performance.
• Troubleshoot Databricks runtime issues, library conflicts, and Spark execution failures.
• Collaborate with cloud and network teams on VNet, peering, and private-link connectivity issues.
• Develop cost governance policies for cluster sizes, job policies, and SQL Warehouse tiers.
• Identify opportunities to reduce cost via autoscaling, spot instances (classic clusters), and job consolidation.

Skills

Databricks administration

Apache Spark programming

PySpark

SQL

Scala

Unity Catalog

Delta Lake

CI/CD for data engineering

Git

Azure

IAM / security patterns

RBAC / SCIM provisioning

Tools

Databricks workspace management

Delta Lake tooling

Unity Catalog management

Git & CI/CD pipelines

Azure monitoring (Azure Monitor)

Key Vault & secrets management

Job description

MISSION : We are seeking a highly skilled Databricks expert with hands-on experience in Apache Spark programming and strong knowledge of cloud-based data engineering platforms. The ideal candidate will be responsible for managing, optimizing, and supporting our Databricks environment, ensuring high availability, security, performance, and efficient usage across teams. This role combines platform administration with practical Spark development to support data engineering and analytics workloads.

Key Responsibilities

Databricks Platform Expert

• Manage, configure, and administer Databricks workspaces, Clusters, SQL Warehouses, Serverless, jobs, and workspace objects.
• Implement and manage Unity Catalog, including catalogs, schemas, tables, access controls, and data lineage.
• Optimize cluster policies, auto-scaling strategies, and cost management for Serverless and Classic compute.
• Serve as the SME for Databricks infrastructure, governance, and security best practices.
• Monitor workspace performance, cluster stability, logs, job reliability, and platform health.
• Implement CI/CD pipelines for notebooks, jobs, and Delta Live Tables using Git integration.
• Support user provisioning, access controls (ACLs), secrets management, and workspace SSO.
• Write efficient Spark (PySpark / SQL / Scala) code for ETL, data transformations, and pipeline optimizations.
• Assist data engineering teams with Spark job debugging, performance tuning, and code reviews.
• Build and maintain production-grade pipelines leveraging Delta Lake, Databricks Jobs, and DLT.
• Implement and manage RBAC, SCIM provisioning, AIM, service principals, and cluster access controls.
• Ensure compliance with enterprise data governance, audit, and logging requirements.
• Manage secrets Key Vault and enforce secure credential handling.
• Support audit reports, compliance reviews, and workspace security configuration.
• Monitor job failures, cluster lifecycle performance, and system events using Databricks logs and cloud-native monitoring tools (Azure Monitor).
• Create automated alerts and observability dashboards for platform usage, cost, and performance.
• Troubleshoot Databricks runtime issues, library conflicts, and Spark execution failures.
• Collaborate with cloud and network teams on VNet, peering, and private-link connectivity issues.
• Develop cost governance policies for cluster sizes, job policies, and SQL Warehouse tiers.
• Identify opportunities to reduce cost via autoscaling, spot instances (classic clusters), and job consolidation.

Required Qualifications

• 4–6 years of experience working with Databricks as an administrator or data engineer.
• Strong expertise in Apache Spark programming (PySpark preferred; SQL or Scala is a plus).
• Hands‑on experience with Databricks Jobs, cluster configuration, SQL Warehouses, and Unity Catalog.
• Deep understanding of Delta Lake, ACID transactions, and lakehouse architecture.
• Experience with Git, CI/CD, and DevOps concepts for data engineering workflows.
• Knowledge of cloud platforms ( Azure).
• Familiarity with IAM, networking basics, monitoring tools, and security patterns