Lead Databricks Data Engineer

Persistent; Pune District; Hybrid; INR 4,000,000 - 8,000,000; Full time

Benefits offered by this job

Competitive salary

Education sponsorship

Quarterly growth opportunities

Flexible work hours

Long Service awards

Annual health check-ups

Group life insurance

Job summary

Persistent is seeking a Lead Databricks Data Engineer to spearhead enterprise data platforms and modern lakehouse architectures across all Persistent locations. The role emphasizes hands-on design, development, and optimization of scalable data pipelines using Databricks, PySpark, Python and SQL.

Ideal candidates have 12–15 years of experience, strong Azure/AWS integration skills, and a track record in AI-powered analytics and governance.

Qualifications

• Lead design, develop, and optimize enterprise-scale data platforms using Databricks.
• Hands-on implementation of cloud-native data architectures and lakehouse patterns.
• Experience with Delta Lake, Unity Catalog and Databricks SQL.
• Strong focus on performance tuning and scalable data pipelines.
• Proven leadership and mentoring of engineering teams.
• Collaboration with data architects, scientists, and stakeholders.

Responsibilities

• Design and maintain scalable data pipelines with Databricks, PySpark, Python, and SQL.
• Build batch and real-time processing frameworks for high-volume data.
• Develop robust ETL/ELT solutions for varied data types.
• Implement data quality, monitoring, and governance frameworks.
• Lead initiatives to improve scalability, reliability, and performance.
• Define standards for Databricks implementations and architecture reviews.
• Develop AI-driven analytics using Databricks Genie and enable self-service insights.

Skills

Databricks

PySpark

Python

SQL

AI-powered analytics

Delta Lake

Unity Catalog

Databricks Workflows

Spark tuning

Leadership

Tools

AWS S3

AWS Glue

AWS Lambda

Amazon EMR

Amazon Redshift

AWS IAM

AWS CloudWatch

Job description

About Position:

We are seeking a highly experienced Lead Databricks Data Engineer with strong expertise in Databricks, PySpark, Python, SQL, and AI-powered analytics solutions. The ideal candidate will lead the design, development, and optimization of enterprise-scale data platforms, modern Lakehouse architectures, and AI-driven analytics solutions. This role requires hands-on experience in building scalable data pipelines, implementing cloud-native data architectures, leveraging Databricks AI capabilities such as Genie, and driving enterprise data modernization initiatives. The successful candidate will work closely with Data Architects, Data Scientists, Product Owners, and business stakeholders to deliver innovative data solutions that enable intelligent decision-making and business growth.

• Role: Lead Databricks Data Engineer
• Location: All Persistent Locations
• Experience: 12 to 15 Years
• Job Type: Full Time Employment

What You'll Do:

• Design, develop, and maintain scalable enterprise-grade data pipelines using Databricks, PySpark, Python, and SQL.
• Build and optimize batch and real-time data processing frameworks for high-volume data workloads.
• Develop robust ETL/ELT solutions for structured, semi-structured, and unstructured data.
• Implement data quality, validation, reconciliation, and monitoring frameworks.
• Establish reusable components, standards, and best practices for enterprise data engineering.
• Lead initiatives focused on scalability, reliability, and performance optimization of data platforms.
• Design and implement modern Databricks Lakehouse architectures.
• Work extensively with Delta Lake, Unity Catalog, Databricks Workflows, and Databricks SQL.
• Tune Spark jobs and optimize cluster configurations for large-scale processing workloads.
• Implement enterprise-grade security, governance, lineage, and access management controls.
• Define architectural standards and best practices for Databricks implementations.
• Develop AI-driven analytics and conversational data solutions using Databricks Genie.
• Enable self-service analytics through natural language interactions with enterprise data.
• Design and implement AI-powered reporting and business intelligence solutions.
• Collaborate with business users to identify and deliver AI-driven analytics use cases.
• Integrate Databricks AI capabilities to improve decision-making and operational efficiency.
• Support Generative AI, intelligent analytics, and advanced data access initiatives.
• Integrate data from enterprise applications, databases, APIs, cloud platforms, and third-party systems.
• Work extensively with AWS services including Amazon S3, AWS Glue, AWS Lambda, Amazon EMR, Amazon Redshift, AWS IAM, AWS CloudWatch.
• Build scalable cloud-native data architectures supporting modern analytics workloads.
• Drive enterprise cloud data modernization initiatives.
• Provide technical leadership to data engineering teams.
• Mentor and coach junior engineers and promote engineering excellence.
• Collaborate with Data Architects, Data Scientists, Product Owners, and business stakeholders.
• Participate in architecture reviews, design discussions, and strategic planning initiatives.
• Lead code reviews and ensure adherence to coding standards and best practices.
• Support troubleshooting, performance optimization, and production issue resolution.
• Actively contribute to innovation initiatives involving AI-powered analytics and modern data platforms.

Expertise You'll Bring:

• 12 to 15 years of experience in Data Engineering, Data Platform Engineering, or Data Warehousing.
• Strong hands-on expertise in Databricks platform implementation and administration.
• Advanced experience with Databricks, PySpark, Python, SQL.
• Proven experience designing and delivering enterprise-scale data engineering solutions.
• Expertise in Databricks Lakehouse architecture and modernization programs.
• Strong experience with Delta Lake, Unity Catalog, Databricks SQL, Databricks Workflows.
• Experience building high-performance ETL/ELT frameworks and distributed data processing pipelines.
• Deep understanding of data modeling, data architecture, and enterprise data management principles.
• Hands-on experience with Databricks Genie and AI-powered analytics capabilities.
• Experience implementing conversational analytics and intelligent data access solutions.
• Strong expertise in performance tuning, query optimization, and Spark workload optimization.
• Experience implementing data governance, security, lineage, and compliance frameworks.
• Strong stakeholder management and leadership capabilities.
• Experience working in Agile delivery environments.
• Excellent communication, presentation, and problem-solving skills.

Benefits:

• Competitive salary and benefits package
• Culture focused on talent development with quarterly growth opportunities and company-sponsored higher education and certifications
• Opportunity to work with cutting-edge technologies
• Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
• Annual health check-ups
• Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents

Values-Driven, People-Centric & Inclusive Work Environment:

Persistent is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds.

• We support hybrid work and flexible hours to fit diverse lifestyles.
• Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.
• If you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment

Let's unleash your full potential at Persistent-persistent.com/careers

\"Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind.\"