Senior Azure Databricks Engineer

NewVision Software; Pune District; Híbrido; INR 1.800.000 - 3.000.000; Jornada completa

Descripción de la vacante

NewVision Softcom & Consultancy Pvt. Ltd. in Pune is seeking a Senior Azure Databricks Engineer to lead end-to-end data pipelines and platform operations on Azure Databricks.

You will work on developing Python/PySpark solutions, managing clusters, and supporting production workloads. The ideal candidate brings 5+ years in data engineering or cloud data platforms with at least 3 years on Azure Databricks, plus strong Python, Spark SQL, Delta Lake, and CI/CD experience.

Formación

• 5+ years of data engineering or cloud data platforms experience.
• At least 3 years with Azure Databricks.
• Strong Python fundamentals and PySpark.
• Experience with ADLS Gen2, Delta Lake, and Spark SQL.
• CI/CD pipelines, automation, and DevOps practices.
• Knowledge of Azure security and Key Vault.

Responsabilidades

• Design, develop, and optimize data pipelines on Azure Databricks.
• Build notebooks, workflows, and scheduled jobs.
• Configure clusters, policies, autoscaling, and runtimes.
• Manage workspace configuration and platform operations.
• Implement Unity Catalog and data governance.
• Collaborate with Data Engineering, DevOps, and Infra teams.

Conocimientos

Python

Azure Databricks

PySpark

Spark SQL

Data governance

CI/CD

Herramientas

Databricks notebooks

Job scheduling

Unity Catalog

Terraform

Descripción del empleo

Company Name New Vision Softcom & Consultancy Pvt. Ltd

Function Business Units (BU)

Department/Practice AI & Analytics

Region APAC

Country India

Base office location Pune

Working model Hybrid

Weekly Off Pune office Standard

State Maharashtra

Skills

Skill

CERTIFICATION

Working Language

Job Description

About the role:

We are looking for a Senior Azure Databricks Engineer with strong hands-on experience in both Databricks development and platform operations.

The ideal candidate should have a good end-to-end understanding of Databricks, from developing Python/PySpark solutions to configuring clusters, managing jobs, and supporting production environments.

Responsibilities

Design, develop, and optimize data pipelines using Azure Databricks, Python, PySpark, Spark SQL, and Delta Lake.

Develop and maintain Databricks notebooks, workflows, and scheduled jobs.

Configure and manage Databricks clusters, cluster policies, autoscaling, runtime versions, and libraries.

Support Databricks workspace configuration, infrastructure management, and platform operations.

Implement Unity Catalog access controls, data governance, and data lineage.

Support Azure security, Key Vault integration, and secrets management.

Implement CI/CD pipelines and automate Databricks deployments.

Monitor and troubleshoot Databricks jobs, clusters, connectivity, and production issues.

Optimize Spark performance, compute resources, and platform costs.

Collaborate with Data Engineering, DevOps, and Infrastructure teams.

Requirements

5+ years of experience in Data Engineering or cloud data platforms, including at least 3 years with Azure Databricks.

Strong Python fundamentals, including data structures, functions, error handling, and problem-solving.

Strong hands-on experience with Azure Databricks, PySpark, Spark SQL, SQL, ADLS Gen2, and Delta Lake.

Practical experience developing Databricks notebooks, workflows, and job scheduling.

Hands-on experience with Databricks cluster management, autoscaling, runtime configuration, and cluster policies.

Experience with Unity Catalog, access management, and data lineage implementation and monitoring.

Good understanding of Databricks infrastructure, workspace configuration, and Azure cloud services.

Experience with CI/CD pipelines, deployment automation, and DevOps practices.

Knowledge of Azure networking, security, Key Vault, and secrets management.

Experience troubleshooting production Databricks jobs, clusters, and platform issues.

Experience with Terraform for provisioning, configuring, and managing Azure Databricks infrastructure, including workspaces, clusters, Unity Catalog, and related Azure resources.

Good understanding of Spark architecture and performance optimization.

Strong communication and problem-solving skills.

Nice to Have

Experience with Bicep, ARM Templates, or other Infrastructure as Code tools.

Experience with Databricks Asset Bundles (DABs).

Experience with Azure Monitor, Log Analytics, and cost optimization.

Experience provisioning Databricks workspaces from scratch.

Experience in banking, insurance, or other regulated industries.