Data Engineer

Hexaware Technologies; India; On-site; INR 1,200,000 - 1,600,000; Full time

Job summary

Hexaware Technologies is seeking a skilled data engineer to drive Azure Databricks development and PySpark-based data transformation. The role focuses on SQL querying, Delta Lake, and Lakehouse concepts to optimize data processing and analytics.

You will implement data pipelines with ADF, manage ADLS Gen2 storage, and collaborate across teams to ensure robust ETL solutions. The ideal candidate will have hands-on experience with Spark SQL, DataFrames, and Spark optimization techniques, plus

Qualifications

• Strong experience in Azure Databricks development.
• Expertise in PySpark for data transformation and processing.
• Advanced knowledge of SQL for querying, optimization, and data analysis.
• Experience with Azure Data Factory (ADF).
• Strong understanding of Delta Lake, Data Lake, and Lakehouse concepts.
• Experience with Azure Data Lake Storage (ADLS Gen2).
• Knowledge of Spark SQL, DataFrames, and Spark optimization techniques.
• Hands-on experience with Data Modeling and ETL frameworks.
• Familiarity with Git, Azure DevOps, and CI/CD pipelines.

Responsibilities

• Develop and optimize data transformation pipelines in Azure Databricks.
• Design and implement SQL-based data queries and performance optimizations.
• Build and maintain data pipelines using ADF and related tools.
• Work with Delta Lake and Lakehouse concepts for reliable data lakes.
• Collaborate with teams to model data and integrate ETL processes.
• Manage version control and CI/CD workflows using Git and Azure DevOps.

Skills

PySpark

SQL

Spark SQL

DataFrames

Delta Lake

Lakehouse concepts

Data Modeling

ETL frameworks

Tools

Azure Databricks

Azure Data Factory (ADF)

ADLS Gen2

Git

Azure DevOps

CI/CD pipelines

Job description

Strong experience in Azure Databricks development.

• Expertise in PySpark for data transformation and processing.
• Advanced knowledge of SQL for querying, optimization, and data analysis.
• Experience with Azure Data Factory (ADF).
• Strong understanding of Delta Lake, Data Lake, and Lakehouse concepts.
• Experience with Azure Data Lake Storage (ADLS Gen2).
• Knowledge of Spark SQL, DataFrames, and Spark optimization techniques.
• Hands-on experience with Data Modeling and ETL frameworks.
• Familiarity with Git, Azure DevOps, and CI/CD pipelines.