Azure Databricks Platform Engineer

NewVision; Pune District; Híbrido; INR 900.000 - 1.500.000; Jornada completa

Descripción de la vacante

New Vision Softcom & Consultancy Pvt. Ltd is seeking a Senior Azure Databricks Platform Engineer to design, develop, and operate data pipelines on Azure Databricks, leveraging Python, PySpark, Spark SQL and Delta Lake.

You will own end-to-end platform activities, including workspace setup, CI/CD deployment, and cluster configuration, in a hybrid Pune environment. The role requires 5+ years in Azure data platforms with at least 3 years in Azure Databricks, strong scripting skills, Unity Catalog

Formación

• 5+ years in Azure data platform engineering.
• 3+ years with Azure Databricks platform management.
• End-to-end Databricks experience: notebooks, jobs, clusters.
• Strong Python fundamentals.
• Understanding of Unity Catalog, access management, and data lineage.

Responsabilidades

• Design, develop, and optimize data pipelines on Azure Databricks.
• Develop and maintain Databricks notebooks and workflows.
• Configure Databricks clusters, policies, autoscaling, runtimes.
• Implement CI/CD pipelines for Databricks deployments.
• Monitor and troubleshoot production Databricks jobs and clusters.

Conocimientos

Azure Databricks platform management

Python/PySpark

Spark SQL

Delta Lake

CI/CD pipelines

DevOps practices

Azure cloud services

Unity Catalog

Cluster management

Monitoring

Herramientas

Terraform

Azure Monitor

Log Analytics

ARM Templates

Bicep

Descripción del empleo

Company Name New Vision Softcom & Consultancy Pvt. Ltd

Function Business Units (BU)

Department/Practice AI & Analytics

Region APAC

Country India

Base office location Pune

Working model Hybrid

Weekly Off Pune office Standard

State Maharashtra

Skills

Skill

Graduation/Equivalent Course

CERTIFICATION

Working Language

Job Description

About the role:

We are looking for a Senior Azure Databricks Platform Engineer with strong hands-on experience in both Databricks development and platform operations. The candidate should be able to manage and support all the activities related to Databricks platform in conjunction with Azure cloud single handedly. Workspace creation, maintenance, deployment using DEVOPS CICD is integral for the role.

Responsibilities

Design, develop, and optimize data pipelines using Azure Databricks, Python, PySpark, Spark SQL, and Delta Lake.

Develop and maintain Databricks notebooks, workflows, and scheduled jobs.

Configure and manage Databricks clusters, cluster policies, autoscaling, runtime versions, and libraries.

Support Databricks workspace configuration, infrastructure management, and platform operations.

Implement Unity Catalog access controls, data governance, and data lineage.

Support Azure security, Key Vault integration, and secrets management.

Implement CI/CD pipelines and automate Databricks deployments.

Monitor and troubleshoot Databricks jobs, clusters, connectivity, and production issues.

Optimize Spark performance, compute resources, and platform costs.

Collaborate with Data Engineering, DevOps, and Infrastructure teams.

Must have Requirements

5+ years of experience in Platform Engineering on Azure data platforms, including at least 3 years with Azure Databricks platform management.

The ideal candidate should have a good end-to-end understanding of Databricks, from developing Python/PySpark solutions to configuring clusters, managing jobs, and supporting production environments. Workspace management, cluster management, Unity catalog implementation.

Strong Python fundamentals, including data structures, functions, error handling, and problem-solving.

Strong hands-on experience with Azure Databricks, PySpark, Spark SQL, SQL, ADLS Gen2, and Delta Lake.

Practical experience developing Databricks notebooks, workflows, and job scheduling.

Hands-on experience with Databricks cluster management, autoscaling, runtime configuration, and cluster policies.

Experience with Unity Catalog, access management, and data lineage implementation and monitoring.

Good understanding of Databricks infrastructure, workspace configuration, and Azure cloud services.

Experience with CI/CD pipelines, deployment automation, and DevOps practices.

Knowledge of Azure networking, security, Key Vault, and secrets management.

Experience troubleshooting production Databricks jobs, clusters, and platform issues.

Experience with Terraform for provisioning, configuring, and managing Azure Databricks infrastructure, including workspaces, clusters, Unity Catalog, and related Azure resources.

Good understanding of Spark architecture and performance optimization.

Strong communication and problem-solving skills.

Nice to Have

Experience with Bicep, ARM Templates, or other Infrastructure as Code tools.

Experience with Databricks Asset Bundles (DABs).

Experience with Azure Monitor, Log Analytics, and cost optimization.

Experience provisioning Databricks workspaces from scratch.

Experience in banking, insurance, or other regulated industries.