Walk-in | Data Engineer

Hectadata India Technology Consulting; Hyderabad; Hybrid; INR 1,200,000 - 1,800,000; Full time

Job summary

Hectadata India Technology Consulting is seeking a data engineer skilled in Azure Databricks and PySpark to design scalable data pipelines and implement robust ETL/ELT processes. The role focuses on building with ADLS Gen2, Delta Lake, and Spark, optimizing queries and pipelines for large volumes, and collaborating with analytics teams to deliver trusted data assets.

Candidates should be proficient in Python, SQL, and cloud data services, and ready to work in an Agile environment with CI/CD

Qualifications

• Design, develop and maintain scalable data engineering solutions using Azure Databricks and PySpark.
• Write complex PySpark code for large-scale data processing and transformation.
• Develop ETL/ELT pipelines with Databricks notebooks and Spark.
• Optimize Spark jobs, notebooks, queries, and pipelines for performance and scalability.

Responsibilities

• Design, develop, and maintain scalable data engineering solutions using Azure Databricks and PySpark.
• Develop and optimize PySpark applications for large-scale data processing and transformation.
• Build and maintain data pipelines using Azure Data Factory (ADF) and Databricks.
• Work with Azure Data Lake Storage (ADLS Gen2) for storing and processing structured and unstructured data.
• Write complex SQL queries for data extraction, transformation, validation, and analysis.
• Develop ETL/ELT workflows using Databricks notebooks and Spark.
• Perform data cleansing, transformation, aggregation, and validation using PySpark.
• Collaborate with data architects, analysts, developers, and business teams to understand data requirements.

Skills

Azure Databricks

PySpark

Python

SQL

Azure Data Factory

Azure Data Lake Storage

Spark

Job description

Mandatory Skills: Azure Databricks, PySpark, Python, SQL, Azure Data Factory, Azure Data Lake Storage (ADLS), Spark

Job Description:

• Design, develop, and maintain scalable data engineering solutions using Azure Databricks and PySpark.
• Develop and optimize PySpark applications for large-scale data processing and transformation.
• Build and maintain data pipelines using Azure Data Factory (ADF) and Databricks.
• Work with Azure Data Lake Storage (ADLS Gen2) for storing and processing structured and unstructured data.
• Write complex SQL queries for data extraction, transformation, validation, and analysis.
• Develop ETL/ELT workflows using Databricks notebooks and Spark.
• Perform data cleansing, transformation, aggregation, and validation using PySpark.
• Optimize Spark jobs, notebooks, queries, and data pipelines for performance and scalability.
• Implement incremental data loads and handle large-volume datasets.
• Work with different data formats such as Parquet, JSON, CSV, and Delta Lake.
• Develop and maintain Delta Lake tables and implement concepts such as partitioning and schema evolution.
• Troubleshoot production data pipelines and resolve performance and data-quality issues.
• Collaborate with data architects, analysts, developers, and business teams to understand data requirements.
• Follow coding standards, security practices, CI/CD processes, and Agile methodologies.