Azure Data Engineer

InfoBeans Inc.; Indore District, Pune District; Hybrid; INR 3,000,000 - 5,400,000; Full time

Job summary

InfoBeans Inc. seeks a Senior Data Engineer for Azure Databricks to design and implement scalable data pipelines and Lakehouse architectures. You will work with PySpark, Spark SQL and Delta Lake to deliver reliable batch and streaming ETL/ELT processes.

Join a team focused on data governance, Azure integration and production-grade data products. Strong Python and SQL skills are required, with 8+ years in software/data engineering and hands-on Databricks experience.

Qualifications

• 8+ years of software/data engineering experience.
• Strong hands-on experience in Azure Databricks and PySpark-based data engineering.
• Strong Python and SQL development skills.
• Experience designing and supporting production-grade data pipelines.
• Familiarity with Delta Lake, Unity Catalog and data governance.

Responsibilities

• Design, develop and maintain scalable data pipelines using Azure Databricks, PySpark and Spark SQL.
• Build Lakehouse solutions with Bronze/Silver/Gold architecture and batch/streaming ETL/ELT pipelines.
• Develop and orchestrate Databricks notebooks, jobs and workflows.
• Integrate Databricks with Azure services like ADLS Gen2, Data Factory, Synapse.
• Optimize Spark workloads with partitioning, caching and shuffle optimization.
• Collaborate with engineering and analytics teams to deliver reliable data products.

Skills

Azure Databricks

PySpark

Spark SQL

Delta Lake

Unity Catalog

Azure data services

Python

SQL

ETL/ELT pipelines

Data governance

Tools

Git

Azure DevOps

Power BI

REST API

Job description

Senior Data Engineer Azure Databricks

Python | PySpark | Spark | Delta Lake | Unity Catalog | Azure

Job Overview

We are looking for an experienced Senior Data Engineer with strong hands‑on expertise in Azure Databricks, PySpark, Apache Spark, Delta Lake and Azure data services. The role will focus on designing and developing scalable data engineering solutions, building batch and streaming pipelines, implementing Lakehouse architectures, and optimizing Spark workloads for performance, reliability and maintainability.

Key Responsibilities

• Design, develop and maintain scalable data pipelines using Azure Databricks, PySpark and Spark SQL. Build and manage Lakehouse solutions using Bronze, Silver and Gold / Medallion architecture. Develop robust batch and streaming ETL/ELT pipelines for structured and semi‑structured data.
• Work extensively with Delta Lake, including MERGE/upserts, schema evolution, data quality and incremental processing.
• Implement and manage data governance using Unity Catalog, including catalogs, schemas, permissions and access controls.
• Develop and orchestrate Databricks notebooks, jobs and workflows for reliable data processing.
• Integrate Databricks with Azure services such as ADLS Gen2, Azure Data Factory, Synapse Analytics, Event Hubs and Service Bus.
• Optimize Spark workloads using appropriate partitioning, caching, broadcast joins, shuffle optimization and other performance‑tuning techniques.
• Apply Spark fundamentals such as lazy evaluation, transformations/actions, joins, repartitioning and data‑skew handling.
• Design data models and curated datasets for analytics, reporting and downstream applications. Implement data quality checks, schema validation, monitoring, error handling and SLA‑driven pipeline delivery. Develop solutions using Python and SQL and follow clean, modular and maintainable coding practices. Use Git, Azure DevOps and CI/CD practices for source control, deployment and release management.
• Collaborate with engineering, analytics and business teams to understand requirements and deliver reliable data products.

Required Technical Skills

Area

Technologies / Expertise

Python, SQL

Apache Spark, PySpark, Spark SQL

Azure Databricks, notebooks, jobs/workflows, performance optimization

Delta Lake, Medallion architecture, incremental processing, MERGE, schema evolution

Unity Catalog, access control, data governance

ADLS Gen2, Azure Data Factory, Azure Synapse Analytics

Azure Event Hubs, Azure Service Bus

Git, Azure DevOps, CI/CD, Docker

Area

Technologies / Expertise

SQL Server, PostgreSQL, Oracle, Azure SQL

Spark & Databricks Expertise

• Strong understanding of Spark execution concepts, including lazy evaluation, transformations and actions. Hands‑on experience with joins, broadcast joins, shuffles, partitioning, repartitioning and caching. Experience troubleshooting and optimizing slow or resource‑intensive Spark jobs.
• Hands‑on Delta Lake experience, including MERGE/upserts, schema evolution and transactional data processing. Practical experience with Unity Catalog and governed access to data assets.
• Ability to implement reliable deduplication, incremental‑load and latest‑record processing patterns using PySpark. Good to Have
• Experience with Azure Functions, Azure Logic Apps, Azure Event Grid, Azure Monitor and Log Analytics. Experience with Apache Kafka or other real‑time streaming technologies.
• Experience with Power BI and REST API‑based data integration.
• Exposure to AI/ML data pipelines, vector search or RAG‑related data preparation.
• Financial services, banking, insurance or risk‑management domain experience.
• Databricks Data Engineer Associate / Professional certification or equivalent practical expertise. Qualifications & Experience
• 8+ years of overall experience in software/data engineering.
• Strong hands‑on experience in Azure Databricks and PySpark‑based data engineering.
• Strong Python and SQL development skills.
• Demonstrated experience designing and supporting production‑grade data pipelines.
• Strong analytical, troubleshooting and problem‑solving skills.
• Good communication and ability to explain technical approaches clearly.