Senior Data Engineer

Kumaran Systems; Chennai; Hybrid; INR 900,000 - 1,600,000; Full time

Job summary

Kumaran Systems is seeking a Senior Data Engineer in Chennai for designing and building large-scale data platforms. The role emphasizes Azure Databricks, PySpark, Python, Scala, SQL, and Delta Lake with hybrid work mode (3 days in-office).

Requires 7+ years of IT experience and 5+ years in relevant data engineering areas. You will collaborate with architects and stakeholders to deliver reliable, secure, high-performance data solutions for analytics, BI, and reporting initiatives.

Qualifications

• 7+ years overall IT experience with 5+ years in Azure Databricks
• Strong Python, PySpark, Scala, SQL skills
• Experience with Delta Lake, Azure Data Lake, Unity Catalog
• ETL/ELT design and data governance basics
• CI/CD and data pipeline orchestration (Airflow/ADF)

Responsibilities

• Design, develop, and optimize enterprise-scale data pipelines using Azure Databricks, PySpark, Python and Scala
• Implement ETL/ELT workflows for structured and semi/unstructured data
• Optimize performance and ensure data quality and security
• Collaborate with stakeholders, architects, and teams to deliver data solutions

Job description

Role: Senior Data Engineer

About Kumaran Systems

Kumaran Systems is a global technology solutions provider delivering innovative IT services and digital transformation solutions to clients worldwide. We foster a collaborative, growth-driven work culture and offer employees opportunities to work on global assignments, enabling both professional and personal development.

Role Overview

We are seeking an experienced Senior Data Engineer to design, develop, and optimize enterprise-scale data platforms and pipelines. The ideal candidate will have strong expertise in Azure Databricks, PySpark, Python, Scala, SQL, and Delta Lake, with hands-on experience in building scalable data solutions and modernizing enterprise data ecosystems.

The role involves collaborating with business stakeholders, architects, and development teams to build reliable, secure, and high-performance data solutions that support analytics, reporting, and business intelligence initiatives.

Location: Guindy, Olympia Technology Park, Chennai - 600032, TN, India

Work Mode: Hybrid (3 Days Work from Office)

Experience Required: 7+ Years

Notice Period: Immediate to 15 Days

Employment Type: Full-Time

Key Responsibilities

Data Pipeline Development

• Design and develop scalable data pipelines using Azure Databricks, Python, PySpark, and Scala.
• Implement ETL/ELT workflows for structured, semi-structured, and unstructured data.
• Develop and maintain data processing workflows using Delta Lake architecture.
• Ensure efficient data ingestion, transformation, and loading processes across enterprise systems.

Data Management & Optimization

• Optimize performance of data pipelines and processing workloads.
• Ensure data quality, reliability, and consistency across data platforms.
• Write, troubleshoot, and optimize complex SQL queries for data processing and validation.
• Implement data governance, security, and access control mechanisms.

Cloud & Data Platform Integration

• Work with Azure Data Lake for storing and managing large-scale datasets.
• Manage data cataloging, governance, and access management using Unity Catalog.
• Support scalable, secure, and high-performing cloud-based data architectures.
• Integrate data engineering solutions with enterprise cloud ecosystems.

Collaboration & Migration Projects

• Work closely with cross-functional teams to deliver data-driven solutions.
• Participate in data migration and modernization initiatives.
• Troubleshoot and resolve data pipeline and platform-related issues.
• Document data architectures, workflows, and operational procedures.

Required Skills & Experience

• Minimum 7 years of overall IT experience with at least 5+ years of hands-on experience in Azure Databricks.
• Strong programming experience in Python, PySpark, and Scala.
• Experience designing and developing scalable data pipelines using Azure Databricks and Delta Lake.
• Good understanding of RDBMS concepts and strong expertise in complex SQL development.
• Preferred experience with Oracle/PLSQL.
• Hands-on experience implementing ETL/ELT solutions for structured, semi-structured, and unstructured data.
• Experience optimizing data processing workloads and ensuring data quality standards.
• Hands-on experience with Azure Data Lake and Unity Catalog.
• Experience integrating data solutions with CI/CD pipelines.
• Experience with orchestration tools such as Apache Airflow or Azure Data Factory is an added advantage.
• Experience working on data migration and modernization projects.
• Good communication and stakeholder management skills.
• Experience with Informatica is preferred but not mandatory.

Preferred Skills

• Experience working on enterprise-scale data engineering projects.
• Strong understanding of data warehousing concepts and best practices.
• Knowledge of cloud-native architecture patterns on Microsoft Azure.
• Familiarity with DevOps practices and CI/CD implementation.
• Experience mentoring junior engineers and leading technical discussions.
• Ability to drive technical solutions independently while collaborating with cross-functional teams.

Relevant Experience Preferred

• Hands-on experience developing scalable data pipelines using Azure Databricks and PySpark.
• Experience implementing ETL/ELT solutions for enterprise data platforms.
• Experience optimizing large-scale data processing workloads and workflows.
• Experience working with Azure Data Lake and cloud-based data architectures.
• Experience supporting data migration, modernization, and transformation initiatives.

Qualifications

Bachelor's or Master's Degree in Computer Science, Information Technology, Engineering, Data Science, or a related field.