Senior Data Engineer

Kumaran Systems; Hyderabad; On-site; INR 4,000,000 - 6,000,000; Full time

Job summary

Kumaran Systems is seeking a qualified Data Engineer in Hyderabad to design, build, and optimize Azure-based data pipelines and lakehouse solutions. You will deliver secure, reliable datasets with governance, automation, and documentation while collaborating in an Agile environment.

The role requires strong expertise in Databricks, Python, Spark, ADF, Delta Lake, and Azure ecosystem, with emphasis on performance, scalability, and data quality. 8+ years of experience is expected.

Qualifications

• 8+ years of experience in Data Engineering.
• Proficiency with Azure Databricks, Python, Spark and Pandas.
• Strong SQL skills and data modeling in star/snowflake schemas.
• Experience with Delta Lake, Unity Catalog, and data governance.
• Hands-on with CI/CD for data pipelines using Azure DevOps or GitHub.
• Familiarity with ADLS and secure data access controls.
• Experience in Agile/Scrum environments and collaboration.

Responsibilities

• Design and build scalable Azure-based data pipelines and lakehouse architectures.
• Develop ETL/ELT workflows from diverse data sources with performance focus.
• Implement data governance, validation, and observability across pipelines.
• Create documentation and data contracts for pipelines and datasets.
• Mentor junior team members and participate in Agile ceremonies.

Skills

Azure Databricks

Python

Spark

SQL

Data Modeling

Delta Lake

CI/CD

Azure Data Factory

Azure Data Lake Storage

Data Governance

Agile/Scrum

ETL/ELT

Tools

Azure DevOps

GitHub Enterprise

Databricks

Jira/Confluence

Job description

We are seeking a skilled Data Engineer to design, build, and optimize Azure-based data pipelines and lakehouse solutions. The ideal candidate will deliver secure, reliable datasets with strong governance, automation, and documentation while collaborating across teams in an Agile environment. This role requires strong expertise in Databricks, Python, Spark, data pipelines, and Azure ecosystem, along with a focus on performance, scalability, and data quality.

Shift Time: 1:00 PM 10:00 PM

Key Responsibilities

• Design and develop scalable data pipelines using Azure and Databricks
• Build and maintain ETL/ELT workflows from diverse data sources
• Experience with Unity Catalog migration and data governance.
• Develop data transformations using Python, Spark, and Pandas
• Implement Delta Lake and Medallion architecture (Bronze/Silver/Gold layers)
• Design and optimize SQL queries and data models (star/dimensional schema)
• Implement CI/CD pipelines for data workflows using Azure DevOps or GitHub
• Ensure data quality, validation, monitoring, and observability
• Maintain clear documentation for pipelines and data contracts
• Participate in Agile ceremonies, code reviews, and team collaboration
• Support and mentor junior team members.

Required Skills & Experience

• 8+ years of experience in Data Engineering
• Strong experience with Azure Databricks (Python, Spark, Pandas)
• Experience with Azure Data Factory (ADF) for orchestration
• Hands-on experience with Delta Lake and Medallion architecture
• Strong knowledge of Azure Data Lake Storage (ADLS)
• Experience with Unity Catalog migration and data governance
• Strong proficiency in SQL and performance optimization
• Experience in ETL/ELT and data modeling (star/dimensional schema)
• Experience with CI/CD tools (Azure DevOps or GitHub Enterprise)
• Understanding of Azure Entra ID for security and access control
• Experience in data validation, testing, and monitoring pipelines
• Familiarity with Agile/Scrum methodologies (Jira/Confluence)
• Strong communication, documentation, and collaboration skills.

Preferred Experience

• Experience with Databricks DevOps (cluster configuration, secrets management)
• Experience with Azure Functions (Python or C#)
• Exposure to Synapse SQL pools, dbt, or Delta Live Tables
• Ability to work independently and contribute to team outcomes
• Strong focus on secure, production-grade data solutions
• Commitment to data reliability, performance, and governance
• Continuous learning mindset and adaptability
• Responsible use of Generative AI tools with validation and guardrails.