Databricks Developer

Persistent Systems; Pune District; Hybrid; INR 2,400,000 - 4,200,000; Full time

Benefits offered by this job

Hybrid work model

Excellent benefits package

Professional development sponsorship

Job summary

Persistent Systems is seeking an experienced Databricks Developer/Lead with 7+ years in data engineering and cloud data platforms. The role focuses on building and optimizing scalable data pipelines using Python and Apache Spark across enterprise data platforms.

You will design, develop, and operate large-scale data workloads on Azure Databricks, manage cloud resources, and collaborate with architects and business teams to deliver data-driven solutions.

Qualifications

• Bachelor's degree in a relevant field.
• 7+ years of experience in data engineering / analytics.
• Proven expertise in Databricks, Spark and distributed data processing.
• Strong Python coding skills and data pipeline development experience.
• Experience with Azure cloud services and containerized deployments.

Responsibilities

• Design, build, and optimize distributed data pipelines using Python and Apache Spark.
• Develop and manage large-scale data processing workloads on Azure Databricks.
• Build scalable ETL/ELT solutions for data ingestion, transformation, and processing.
• Manage data storage and cloud resources using Azure Storage, AKS, and related services.
• Develop container-based deployment solutions for data engineering workloads.
• Orchestrate data workflows using Airflow and enterprise schedulers.
• Implement data security controls including access management and encryption.
• Support vulnerability remediation across data platforms using security tooling.
• Monitor pipelines and troubleshoot performance, reliability, and operational issues.
• Collaborate with data architects and business teams to deliver data solutions.
• Maintain code repositories with GitHub and document in Confluence.
• Participate in code reviews and promote data engineering best practices.
• Support CI/CD processes and deployment automation.
• Contribute to Agile ceremonies and release management.
• Ensure compliance with data governance and security requirements.

Skills

Databricks

Apache Spark

Python

Distributed data processing

Azure cloud

Data pipelines design

Collaboration with stakeholders

Education

Bachelor's degree in Computer Science / IT / Engineering

Tools

Azure Databricks

AKS

Azure Storage

Airflow

GitHub

Confluence

CI/CD tooling

Job description

We are looking for an experienced Databricks Developer / Lead with 7+ years of experience in Data Engineering, Analytics, and Cloud Data Platforms. The ideal candidate will possess deep expertise in Databricks, Apache Spark, distributed data processing, and cloud-native data architectures. This role involves designing, developing, and optimizing large-scale data pipelines while ensuring high performance, scalability, reliability, and operational excellence across enterprise data platforms.

• Role: Databricks Developer
• Location: All Persistent Locations
• Experience: 7 to 12 years
• Job Type: Full-Time Employment

What You'll Do:

• Design, build, and optimize distributed data pipelines using Python and Apache Spark.
• Develop, manage, and maintain large-scale data processing workloads on Azure Databricks.
• Build scalable ETL and ELT solutions for data ingestion, transformation, and processing.
• Manage data storage and cloud resources using Azure Storage, Azure Container Registry (ACR), and Azure Kubernetes Service (AKS).
• Develop container-based deployment solutions for data engineering workloads.
• Orchestrate data workflows using Apache Airflow and enterprise scheduling platforms.
• Implement robust data security controls including access management, encryption, and secure credential handling.
• Support vulnerability remediation activities across data platforms using tools such as CodeQL, Cycode, and related security frameworks.
• Monitor data pipelines and troubleshoot performance, reliability, and operational issues.
• Collaborate with data architects, business analysts, and application teams to deliver business-driven data solutions.
• Maintain source code repositories using GitHub and ensure proper documentation in Confluence.
• Participate in code reviews and enforce data engineering best practices and development standards.
• Support CI/CD processes and deployment automation for data platform components.
• Contribute to Agile ceremonies including sprint planning, development activities, testing, and release management.
• Ensure compliance with organizational data governance, security, and regulatory requirements.

Expertise You'll Bring:

• 6 to 9 years of hands-on experience in Data Engineering and large-scale data processing solutions.
• Extensive experience building distributed data processing pipelines using Apache Spark and PySpark.
• Hands-on experience with Azure Databricks for data transformation and analytics workloads.
• Experience deploying and managing data solutions on Azure Kubernetes Service (AKS).
• Strong knowledge of Azure Storage services, Azure Container Registry (ACR), and cloud-native architectures.
• Experience with Apache Airflow and workflow orchestration frameworks.
• Understanding of ETL/ELT design patterns and data pipeline architecture.
• Experience implementing data security controls, encryption standards, and secure secret management practices.
• Familiarity with vulnerability management tools such as CodeQL, Cycode, or similar security scanning platforms.
• Experience working with GitHub, version control practices, and CI/CD pipelines.
• Strong understanding of containerization technologies and deployment automation.
• Experience troubleshooting and optimizing distributed data processing workloads.
• Strong analytical, debugging, and problem-solving skills.
• Excellent communication and stakeholder collaboration abilities.
• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline.
• Competitive salary and benefits package
• Culture focused on talent development with quarterly growth opportunities and company-sponsored higher education and certifications
• Opportunity to work with cutting-edge technologies
• Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
• Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents

Values-Driven, People-Centric & Inclusive Work Environment:

Persistent is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds.

• We support hybrid work and flexible hours to fit diverse lifestyles.
• Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.
• If you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment

Let's unleash your full potential at Persistent - persistent.com/careers

\"Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind.\"