Azure Data Engineer

Novastrid It Ventures; Bengaluru; On-site; INR 1,400,000 - 2,300,000; Full time

Job summary

Novastrid It Ventures in Bengaluru is seeking an experienced Azure Data Engineer to design, develop, and maintain scalable data pipelines on the Azure cloud, leveraging Azure Databricks, PySpark, Python and SQL. The role focuses on building robust data solutions, optimizing performance and enabling reliable data processing across large datasets.

You will collaborate with data architects and analysts, implement ETL/ELT workflows, ensure data quality, and contribute to CI/CD practices in an Agile

Qualifications

• Must have hands-on experience with designing and building data pipelines on Azure.
• Strong knowledge of Databricks, Spark, PySpark and SQL for data transformation.
• Proficiency in Python and SQL for ETL/ELT processes.
• Experience with data validation, quality checks, and CI/CD practices.

Responsibilities

• Develop and maintain scalable data pipelines using Azure Databricks and PySpark.
• Write efficient Python and SQL code for data extraction, transformation, and loading.
• Perform data transformation and processing with PySpark DataFrames and Spark SQL.
• Design and implement ETL/ELT workflows for structured and semi-structured data.
• Optimize Spark jobs, SQL queries, and Databricks workloads for performance and cost.
• Work with Azure data services to build cloud-based data solutions.
• Perform data validation, cleansing, and quality checks.
• Troubleshoot data pipeline failures and resolve performance issues.
• Collaborate with Data Architects, Analysts, Developers, and business teams.
• Follow coding standards, CI/CD practices, version control, and Agile methodologies.
• Participate in design discussions, code reviews, testing, and production deployments.

Skills

Azure Databricks

PySpark

Python

SQL

Data Engineering / ETL / ELT

Data pipelines

data transformation

data cleansing

data integration

Azure cloud data services

large-volume datasets

Job description

Job Summary

We are looking for an experienced Azure Data Engineer with strong hands‑on expertise in Azure Databricks, PySpark, Python, and SQL. The candidate will be responsible for designing, developing, and maintaining scalable data pipelines and data processing solutions on the Azure cloud platform.

Mandatory Skills

• Azure Databricks
• PySpark
• Python
• SQL
• Data Engineering / ETL / ELT
• Experience in developing and optimizing data pipelines
• Strong understanding of data transformation, data cleansing, and data integration
• Good knowledge of Azure cloud data services
• Experience working with large-volume datasets and distributed processing

Key Responsibilities

• Develop and maintain scalable data pipelines using Azure Databricks and PySpark.
• Write efficient Python and SQL code for data extraction, transformation, and loading.
• Perform data transformation and processing using PySpark DataFrames and Spark SQL.
• Design and implement ETL/ELT workflows for structured and semi-structured data.
• Optimize Spark jobs, SQL queries, and Databricks workloads for performance and cost.
• Work with Azure data services to build cloud-based data solutions.
• Perform data validation, cleansing, and quality checks.
• Troubleshoot data pipeline failures and resolve performance issues.
• Collaborate with Data Architects, Analysts, Developers, and business teams.
• Follow coding standards, CI/CD practices, version control, and Agile methodologies.
• Participate in design discussions, code reviews, testing, and production deployments.