Data Engineer

SWITS DIGITAL Private Limited; Chennai; Remote; INR 1,800,000 - 3,000,000; Full time

Job summary

SWITS DIGITAL Private Limited is seeking a Data Engineer with 8+ years of experience to lead data integration initiatives across Pan India. The role focuses on Azure services, PySpark, Python development, and robust ETL/ELT pipelines while ensuring data governance and quality through Soda and Alation.

You will design scalable pipelines, optimize SQL queries, and collaborate with data analysts and business stakeholders to deliver reliable data solutions with strong security and compliance

Qualifications

• Hands-on experience with Azure cloud services and data engineering
• Proficiency in Python, PySpark, and SQL for data pipelines
• Experience building and optimizing ETL/ELT processes and data processing frameworks
• Familiarity with data governance and data quality tools like Soda and Alation

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure, Python and PySpark.
• Build and optimize ETL/ELT processes for data extraction, transformation and loading.
• Write complex SQL queries and tune database performance.
• Develop data processing solutions with PySpark for large datasets.
• Implement data quality checks and monitoring with Soda.
• Support data cataloging, metadata management and governance using Alation.
• Ensure data accuracy, security and governance compliance.
• Troubleshoot data pipeline failures and improve reliability.
• Collaborate with analysts, architects and stakeholders to deliver reliable data solutions.

Skills

Azure Services

Python

PySpark

ETL/ELT

SQL

Soda

Alation

Education

BE/B.Tech/ME/M.Tech/MCA/M.Sc graduates only

Job description

Job Title: Data Engineer

Experience: 8+ Years

Location: Pan India

Notice Period: Immediate to 15 Days

Education: BE/B.Tech/ME/M.Tech/MCA/M.Sc graduates only

Job Summary

We are looking for an experienced Data Engineer with strong hands-on expertise in Azure Services, Python, PySpark, ETL development, and SQL. The ideal candidate should have experience building and optimizing data pipelines, managing large datasets, and implementing data governance and data quality solutions using tools such as Soda and Alation.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure services, Python, and PySpark.
• Build and optimize ETL/ELT processes for data extraction, transformation, and loading.
• Write complex SQL queries and optimize database performance.
• Develop data processing solutions using PySpark for large-scale datasets.
• Implement data quality checks, monitoring, and validation using Soda.
• Support data cataloging, metadata management, data lineage, and governance using Alation.
• Ensure data accuracy, consistency, security, and compliance with governance standards.
• Troubleshoot data pipeline failures and improve performance and reliability.
• Collaborate with data analysts, architects, and business stakeholders to deliver reliable data solutions.

Essential Skills

• Azure Services
• Python programming
• PySpark
• ETL/ELT development
• SQL
• Data governance and data quality tools such as Soda and Alation