Senior Data Engineer

ThoughtSol Infotech Ltd.; Dadri; On-site; INR 800,000 - 1,200,000; Full time

Job summary

ThoughtSol Infotech Ltd. is seeking a Senior Data Engineer with strong Azure Data Engineering experience to design and implement scalable data pipelines on Microsoft Azure.

You will work on ETL/ELT pipelines with ADF, ADLS, Databricks, Delta Lake, and Synapse, building robust data models and ensuring data quality. Proficiency in PySpark/Scala, Python, and SQL, plus Azure DevOps for CI/CD, is essential.

Qualifications

• 2.5+ years of experience in Data Engineering.
• Hands-on with Azure Data Factory, ADLS, Databricks.
• Proficient in Python and SQL.
• Knowledge of Azure services and cloud concepts.

Responsibilities

• Design, develop, and maintain scalable data pipelines on Microsoft Azure.
• Develop ETL/ELT pipelines using Azure Data Factory.
• Work with ADLS and Azure Databricks and Delta Lake.
• Implement batch and real-time data ingestion and processing.
• Utilize Spark Structured Streaming and Synapse Analytics.
• Build data models and optimize data workflows.
• Collaborate with cross-functional teams and clients.
• Use Python and shell scripting for automation.
• Apply CI/CD and deployment practices via Azure DevOps.
• Troubleshoot performance and data quality issues.

Skills

PySpark/Spark/Scala

Spark Streaming

Batch & real-time processing

Azure SQL / Cosmos DB

Python

Shell scripting

Azure DevOps & CI/CD

SQL & data manipulation

Independent & team work

Communication & problem solving

Tools

Azure Data Factory

Azure Data Lake Storage

Azure Databricks

Delta Lake

Spark Structured Streaming

Azure Synapse Analytics

Cosmos DB

Azure SQL Data Warehouse

Azure DevOps

Job description

We are looking for a skilled and experienced Senior Data Engineer with strong expertise in the Azure Data Engineering ecosystem. The ideal candidate should have hands-on experience in designing and developing scalable data pipelines, batch and real-time data processing, and working with Azure data services. The candidate will be responsible for building robust data solutions, collaborating with cross-functional teams, and contributing to end-to-end data engineering projects.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and data processing solutions on Microsoft Azure.
• Develop and manage ETL/ELT pipelines using Azure Data Factory (ADF).
• Work extensively with Azure Data Lake Storage (ADLS) and Azure Databricks.
• Implement batch and real-time data ingestion and processing solutions.
• Work with Delta Lake and Spark Structured Streaming.
• Design and work with Azure Synapse Analytics / Synapse Workspace.
• Work with Azure SQL Data Warehouse / Azure Synapse and Cosmos DB.
• Develop and maintain data models and optimize data processing workflows.
• Use Python and Shell scripting for automation and data engineering tasks.
• Work with Azure DevOps for source control, CI/CD, and deployment processes.
• Understand and work with basic Azure infrastructure and cloud concepts.
• Participate in end-to-end implementation of data engineering projects.
• Troubleshoot performance, data quality, and pipeline-related issues.
• Collaborate with technical teams, stakeholders, and clients to understand requirements and deliver effective data solutions.

Required Skills

• Minimum 2.5 years of relevant experience in Data Engineering.
• Strong hands-on experience with:
• o PySpark / Spark / Scala
• o Spark Streaming
• Strong understanding of batch and real-time data processing.
• Good knowledge of Azure SQL / SQL Data Warehouse and Cosmos DB.
• Good programming experience in Python.
• Knowledge of Shell scripting.
• Understanding of Azure DevOps, CI/CD, and Azure infrastructure concepts.
• Strong SQL and data manipulation skills.
• Ability to work independently as well as collaboratively in a team environment.
• Strong analytical, problem-solving, and communication skills.

Good to Have

• Azure or other relevant cloud certifications.
• Experience working on an end-to-end Azure Data Engineering project.
• Experience interacting directly with clients or stakeholders.
• Knowledge of data architecture and scalable cloud-based data solutions.