Databricks Developer

Kumaran; Hyderabad; On-site; INR 1,000,000 - 1,400,000; Full time

Job summary

Kumaran is seeking a skilled Databricks Developer in Hyderabad to design and optimize scalable data pipelines using Apache Spark, Python, and SQL.

The ideal candidate will have hands-on experience with Databricks and cloud platforms, and will work closely with data engineers and analysts to support data-driven decision-making.

The role demands strong problem-solving skills and a solid understanding of data governance principles.

Qualifications

• 3–5+ years of experience with Databricks, Apache Spark, and Python (PySpark).
• Strong experience building scalable ETL/ELT pipelines.
• Advanced knowledge of Spark SQL or Databricks SQL for data transformation and analysis.
• Hands-on experience with cloud platforms like Azure or AWS.

Responsibilities

• Design and develop ETL/ELT data pipelines using PySpark and SQL.
• Implement data lake and warehouse architectures using Delta Lake.
• Optimize Spark jobs for performance and cost-efficiency.
• Ensure data quality checks and monitoring processes.

Skills

Databricks

Apache Spark

Python (PySpark)

ETL/ELT Data Pipelines

SQL

Data Governance

Agile Development

Git

Education

Bachelor’s or Master’s degree in Computer Science or related field

Tools

Azure Databricks

AWS

Apache Airflow

Jenkins

Job description

Kumaran Systems: Engineering Future-Ready Transformations Since 1992, Kumaran Systems has been helping enterprises bridge the past and the future, reimagining legacy systems and embracing AI-led possibilities. With over 34 years of delivery excellence, we specialise in re-engineering core applications, driving cloud transformation, and building automation-first, GenAI-enabled ecosystems across the Banking and Financial Services, Insurance, Telecom, and Automotive sectors. We help organisations not just upgrade their systems but reimagine what’s possible.

Why Kumaran?

We blend engineering discipline with AI innovation, helping clients modernise with confidence, automate with clarity, and scale with purpose. Our global delivery model ensures agility, responsiveness, and seamless collaboration, with clients always at the heart of every engagement. At Kumaran, we don’t just solve problems, we engineer future-ready transformations.

Job Description

Job Summary

We are seeking a skilled Databricks Developer to design, develop, and optimize scalable data pipelines and analytics solutions using Apache Spark, Python (PySpark), and SQL within modern cloud environments. The ideal candidate will have hands-on experience working with Databricks, Delta Lake, and cloud data platforms, and will be responsible for building high-performance data processing workflows that support data-driven decision-making.

Key Responsibilities

• Design, develop, and maintain ETL/ELT data pipelines using PySpark and SQL in Databricks notebooks.
• Process large-scale datasets and ensure reliable and efficient data transformations.
• Implement data lake and data warehouse architectures using Databricks Delta Lake and Delta Live Tables.
• Build and manage Medallion Architecture (Bronze, Silver, Gold layers) for structured data processing.
• Integrate Databricks solutions with cloud services such as AWS S3 or other cloud storage platforms.
• Build and manage automated workflows and scheduling using Databricks Jobs, Airflow, or CI/CD pipelines.
• Integrate with DevOps tools such as Azure DevOps or Jenkins for continuous integration and deployment.
• Work closely with data engineers, analysts, and business stakeholders to translate business requirements into scalable data solutions.
• Participate in Agile development processes including sprint planning and technical discussions.

Performance Optimization

• Optimize Spark jobs and queries for performance, scalability, and cost-efficiency.
• Manage cluster configurations, partitioning strategies, and caching mechanisms.

Data Governance & Quality

• Implement data quality checks, validation frameworks, and monitoring processes.
• Ensure data security, encryption, masking, and lineage tracking.

Workflow Automation

• Build and manage automated workflows and scheduling using Databricks Jobs, Airflow, or CI/CD pipelines.
• Integrate with DevOps tools such as Azure DevOps or Jenkins for continuous integration and deployment.

Required Skills & Qualifications

Core Technologies

• 3–5+ years of experience with Databricks, Apache Spark, and Python (PySpark).
• Strong experience building scalable ETL/ELT pipelines.

SQL Expertise

• Advanced knowledge of Spark SQL or Databricks SQL for data transformation and analysis.

Cloud Platforms

• Hands-on experience with Azure Databricks, AWS, or Google Cloud Platform (GCP).

Data Modeling

• Experience with Delta Lake and Medallion Architecture (Bronze/Silver/Gold layers).
• Strong understanding of data modeling and data warehouse concepts.

Version Control & CI/CD

• Proficiency with Git and CI/CD tools such as Azure DevOps or Jenkins.

Preferred Qualifications

• Experience with Data Build Tool (dbt).
• Knowledge of workflow orchestration tools such as Apache Airflow.
• Experience with Machine Learning workflows using MLflow.
• Familiarity with data governance and enterprise data platforms.

Education

• Bachelor’s or Master’s degree in Computer Science, Information Technology, Data Engineering, or a related field.

Strong problem-solving and analytical skills

Ability to handle large-scale data processing environments

Good communication and collaboration skills in Agile teams

Kumaran Systems is an Equal Opportunity Employer and does not discriminate on the basis of race or ethnicity, religion, sex, national origin, age, veteran disability or genetic information or any other reason prohibited by law in employment.