Hiring For Azure Data Engineer

Think People Solutions; Hyderabad, Chennai District, Bengaluru; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Think People Solutions is seeking an AI / Data Engineer to design, develop and optimize scalable data pipelines on Azure. You will leverage Python, PySpark and Databricks to transform large datasets and enable AI/ML workloads.

The role requires hands-on experience with ETL/ELT, SQL and cloud data platforms, with exposure to Azure Synapse Analytics as a plus. You will collaborate with data scientists and analysts in an Agile environment.

Qualifications

• Strong hands-on Python development and data engineering experience.
• Experience with PySpark and Databricks for large-scale data processing.
• Good knowledge of SQL, ETL/ELT patterns and cloud data platforms.

Responsibilities

• Design, develop and maintain scalable ETL/ELT data pipelines using Azure Databricks and PySpark.
• Develop data transformation and processing solutions using Python and PySpark.
• Work with large-scale structured and semi-structured datasets using Apache Spark.
• Build and optimize data pipelines for performance, scalability and reliability.
• Develop Databricks notebooks, jobs and workflows for batch data processing.
• Implement data cleansing, transformation, validation and reconciliation processes.
• Work with Azure data services to build cloud-based data engineering solutions.
• Support development of data platforms for AI/ML and advanced analytics use cases.
• Prepare high-quality datasets/features for AI and Machine Learning models.
• Optimize Spark jobs, queries, partitions and resource utilization.
• Implement data quality, error handling, monitoring and troubleshooting mechanisms.
• Work closely with Data Scientists, Data Analysts, Architects and other engineering teams.
• Participate in code reviews, technical discussions and Agile delivery processes.
• Maintain technical documentation related to data pipelines and data processing solutions.

Skills

Python

PySpark

Azure

Databricks

ETL/ELT

SQL

AI/ML workloads

Tools

Azure Databricks

Databricks notebooks

Job description

We are looking for an experiencedAI / Data Engineerwith strong hands-on expertise inPython, PySpark, Azure and Databricks. The candidate will be responsible for designing, developing and optimizing scalable data pipelines and data processing solutions on Azure.

The ideal candidate should have strong experience inPySpark and Databricks, along with good knowledge of data engineering, ETL/ELT, SQL and cloud data platforms. Exposure toAI/ML workloads and Azure Synapse Analyticswill be an added advantage.

Key Responsibilities

• Design, develop and maintain scalableETL/ELT data pipelinesusing Azure Databricks and PySpark.
• Develop data transformation and processing solutions usingPython and PySpark.
• Work with large-scale structured and semi-structured datasets using Apache Spark.
• Build and optimize data pipelines for performance, scalability and reliability.
• Develop Databricks notebooks, jobs and workflows for batch data processing.
• Implement data cleansing, transformation, validation and reconciliation processes.
• Work with Azure data services to build cloud-based data engineering solutions.
• Support development of data platforms forAI/ML and advanced analytics use cases.
• Prepare high-quality datasets/features for AI and Machine Learning models.
• Optimize Spark jobs, queries, partitions and resource utilization.
• Implement data quality, error handling, monitoring and troubleshooting mechanisms.
• Work closely with Data Scientists, Data Analysts, Architects and other engineering teams.
• Participate in code reviews, technical discussions and Agile delivery processes.
• Maintain technical documentation related to data pipelines and data processing solutions.