Senior Data Engineer

Kumaran Systems; Chennai District; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Kumaran Systems in India seeks an experienced Senior Databricks Developer with 5–8 years in data engineering to design scalable data solutions. You will work on Databricks, Apache Spark, and cloud-based data platforms to deliver robust pipelines and governance.

The role requires hands-on Python, Scala, and SQL skills, plus experience in ETL/ELT, data lakes, data warehouses, and cross-functional collaboration with data scientists and engineers. This is a senior, on-site position in Chennai.

Qualifications

• 5–8 years of experience in data engineering or big data development.
• Hands-on Databricks expertise and Spark processing experience.
• Proficiency in Python, Scala and SQL.
• Experience building scalable ETL/ELT data pipelines.
• Experience with data lake and data warehouse environments.
• Experience on cloud platforms (Azure, AWS or GCP).
• Strong data modelling, architecture and integration knowledge.
• Excellent analytical, troubleshooting and performance optimisation skills.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks.
• Build and optimise large-scale data processing with Apache Spark.
• Develop and manage ETL/ELT workflows for structured and unstructured data.
• Write data processing scripts in Python, Scala, and SQL.
• Implement and manage data lake and data warehouse architectures.
• Optimize performance of pipelines and Spark jobs.
• Integrate Databricks with cloud data services and storage.
• Collaborate with data scientists, analysts, and engineers to deliver reliable data solutions.
• Implement data governance, data quality, and security best practices.
• Troubleshoot pipeline issues and document architectures and processes.

Skills

Databricks

Apache Spark

Python

Scala

SQL

ETL/ELT

Data modelling

Data architecture

Cloud platforms

Troubleshooting

Job description

We are seeking an experienced Senior Databricks Developer with 5 to 8 yrs of experience in designing and implementing scalable data engineering solutions. The ideal candidate will have strong expertise in big data processing, data pipeline development, and cloud-based data platforms, with hands‑on experience using Databricks and distributed data processing frameworks like Apache Spark.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks.
• Build and optimise large-scale data processing solutions using Apache Spark.
• Develop and manage ETL/ELT workflows for ingesting, transforming, and processing structured and unstructured data.
• Write data processing scripts using Python, Scala, and SQL.
• Implement and manage data lake and data warehouse architectures.
• Optimise performance of data pipelines and Spark jobs.
• Integrate Databricks with cloud data services and storage platforms.
• Collaborate with data scientists, analysts, and engineering teams to deliver reliable data solutions.
• Implement best practices for data governance, data quality, and security.
• Troubleshoot data pipeline issues and ensure system reliability and scalability.
• Document data architecture, workflows, and operational processes.

Required Skills & Qualifications

• 5 to 8 years of experience in data engineering or big data development.
• Good hands‑on experience with Databricks.
• Extensive experience with distributed data processing using Apache Spark.
• Proficiency in Python, Scala, and SQL.
• Strong experience building scalable ETL/ELT data pipelines.
• Experience working with large datasets in data lake or data warehouse environments.
• Experience with cloud platforms such as Microsoft Azure, Amazon Web Services, or Google Cloud Platform.
• Strong knowledge of data modelling, data architecture, and data integration concepts.
• Excellent analytical, troubleshooting, and performance optimisation skills.