Data Engineer

ThoughtSol Infotech Ltd.; Dadri; On-site; INR 1,500,000 - 2,300,000; Full time

Job summary

ThoughtSol Infotech Ltd. seeks a Senior Data Engineer with strong Azure Data Engineering expertise to design and implement scalable data pipelines. You will work with ADLS, Databricks, Delta Lake, and Synapse, building robust data solutions for batch and real-time processing.

Responsibilities include developing ETL/ELT pipelines, managing data models, scripting in Python and Shell, and collaborating with cross-functional teams using Azure DevOps for CI/CD and deployments.

Qualifications

• Minimum 2.5 years of relevant experience in Data Engineering.
• Hands-on experience with PySpark, Spark, or Scala.
• Experience with Spark Streaming.
• Strong experience with batch and real-time data processing.
• Knowledge of Azure SQL Data Warehouse and Cosmos DB.
• Python programming experience.
• Shell scripting knowledge.
• Understanding of Azure DevOps, CI/CD, and Azure infrastructure concepts.
• Strong SQL and data manipulation skills.
• Ability to work independently as well as collaboratively in a team.
• Strong analytical, problem-solving, and communication skills.

Responsibilities

• Design, develop, and maintain scalable data pipelines and data processing solutions on Microsoft Azure.
• Develop and manage ETL/ELT pipelines using Azure Data Factory (ADF).
• Work extensively with Azure Data Lake Storage (ADLS) and Azure Databricks.
• Implement batch and real-time data ingestion and processing solutions.
• Work with Delta Lake and Spark Structured Streaming.
• Design and work with Azure Synapse Analytics / Synapse Workspace.
• Work with Azure SQL Data Warehouse / Azure Synapse and Cosmos DB.
• Develop and maintain data models and optimize data processing workflows.
• Use Python and Shell scripting for automation and data engineering tasks.
• Work with Azure DevOps for source control, CI/CD, and deployment processes.
• Understand and work with basic Azure infrastructure and cloud concepts.
• Participate in end-to-end implementation of data engineering projects.
• Troubleshoot performance, data quality, and pipeline-related issues.
• Collaborate with technical teams, stakeholders, and clients to deliver effective data solutions.

Skills

PySpark

Spark

Scala

Batch processing

Real-time processing

Python

Shell scripting

Azure DevOps

SQL

Communication

Tools

Azure Data Factory

Azure Data Lake Storage

Azure Databricks

Delta Lake

Spark Structured Streaming

Azure Synapse Analytics

Cosmos DB

Job description

We are looking for a skilled and experienced Senior Data Engineer with strong expertise in the Azure Data Engineering ecosystem. The ideal candidate should have hands-on experience in designing and developing scalable data pipelines, batch and real-time data processing, and working with Azure data services. The candidate will be responsible for building robust data solutions, collaborating with cross-functional teams, and contributing to end-to-end data engineering projects.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and data processing solutions on Microsoft Azure.
• Develop and manage ETL/ELT pipelines using Azure Data Factory (ADF).
• Work extensively with Azure Data Lake Storage (ADLS) and Azure Databricks.
• Implement batch and real-time data ingestion and processing solutions.
• Work with Delta Lake and Spark Structured Streaming.
• Design and work with Azure Synapse Analytics / Synapse Workspace.
• Work with Azure SQL Data Warehouse / Azure Synapse and Cosmos DB.
• Develop and maintain data models and optimize data processing workflows.
• Use Python and Shell scripting for automation and data engineering tasks.
• Work with Azure DevOps for source control, CI/CD, and deployment processes.
• Understand and work with basic Azure infrastructure and cloud concepts.
• Participate in end-to-end implementation of data engineering projects.
• Troubleshoot performance, data quality, and pipeline-related issues.
• Collaborate with technical teams, stakeholders, and clients to understand requirements and deliver effective data solutions.

Required Skills

• Minimum 2.5 years of relevant experience in Data Engineering.
• Strong hands-on experience with:
• o PySpark / Spark / Scala
• o Spark Streaming
• Strong understanding of batch and real-time data processing.
• Good knowledge of Azure SQL / SQL Data Warehouse and Cosmos DB.
• Good programming experience in Python.
• Knowledge of Shell scripting.
• Understanding of Azure DevOps, CI/CD, and Azure infrastructure concepts.
• Strong SQL and data manipulation skills.
• Ability to work independently as well as collaboratively in a team environment.
• Strong analytical, problem-solving, and communication skills.

Good to Have

• Azure or other relevant cloud certifications.
• Experience working on an end-to-end Azure Data Engineering project.
• Experience interacting directly with clients or stakeholders.
• Knowledge of data architecture and scalable cloud-based data solutions.