Azure Data Engineer - SQL Databricks

Crescendo Global Leadership Hiring India; Dadri, Gurugram District, Bengaluru; Hybrid; INR 1,400,000 - 2,600,000; Full time

Job summary

Crescendo Global Leadership Hiring India is seeking an experienced Data Engineer to join a high-performing data team. You will build robust, scalable data pipelines on Azure, enabling advanced analytics across enterprise platforms.

The role requires 7-10 years of experience, deep hands-on expertise with Azure Data Factory, Databricks, ADLS Gen2, Delta Lake, and PySpark, along with a strong grasp of data governance and cost optimization.

Qualifications

• 7-10 years of data engineering experience, focused on cloud-based transformations.
• Hands-on with Azure data platforms and large-scale ETL/ELT pipelines.
• Strong PySpark, Python, and SQL skills with data quality focus.
• Experience with Delta Lake, Medallion Architecture, and data governance.

Responsibilities

• Design and maintain end-to-end data pipelines using ADF and Databricks.
• Build scalable ETL/ELT workflows for high-volume data.
• Develop PySpark-based data processing frameworks and optimize performance.
• Ingest data from APIs, databases, and files into ADLS Gen2.
• Ensure data quality, transformations, and validation across pipelines.
• Collaborate with analysts, scientists, and stakeholders to deliver data solutions.
• Monitor and optimize pipelines for cost efficiency and reliability.
• Adhere to data governance, security, and compliance standards.

Skills

PySpark

Python

SQL

Data orchestration

CI/CD tooling

Tools

Azure Data Factory

Azure Databricks

Azure Data Lake (ADLS Gen2)

Synapse Analytics

Delta Lake

Medallion Architecture

Azure DevOps

Git

Job description

Hiring: Data Engineer Azure & Databricks

7- 10 years

Gurugram/Bangalore

We are looking for an experienced Data Engineer to join a high-performing data team working on large-scale, cloud-based data transformation initiatives. This role involves building robust, scalable data pipelines and enabling advanced analytics across enterprise data platforms.

Key Responsibilities:

• Design, develop, and maintain end-to-end data pipelines using Azure Data Factory (ADF) and Azure Databricks
• Build scalable ETL/ELT workflows to process high-volume structured and unstructured data
• Develop and optimize PySpark-based data processing frameworks
• Ingest data from multiple sources such as APIs, databases, and flat files into Azure Data Lake
• Implement data transformation, cleansing, and validation logic to ensure high data quality
• Work with Delta Lake architecture for efficient storage and processing
• Collaborate with data analysts, data scientists, and business stakeholders to deliver data solutions
• Monitor, troubleshoot, and optimize data pipelines for performance and cost efficiency
• Ensure adherence to data governance, security, and compliance standards

Technical Skills Required:

• Strong hands-on experience with Azure Data Factory, Azure Databricks
• Expertise in PySpark, Python, SQL
• Experience with Azure Data Lake (ADLS Gen2), Synapse Analytics
• Knowledge of Delta Lake, Medallion Architecture (Bronze/Silver/Gold layers)
• Familiarity with data warehousing concepts (Star Schema, Snowflake Schema)
• Experience with CI/CD tools (Azure DevOps, Git)
• Understanding of data orchestration and workflow scheduling
• Exposure to performance tuning and optimization techniques

Good to Have:

• Experience with streaming data pipelines (Kafka/Event Hub)
• Knowledge of Power BI or other visualization tools
• Exposure to data security and access control (RBAC, IAM)