Senior Databricks Data Engineer

Persistent; Pune District; Hybrid; INR 3,000,000 - 6,000,000; Full time

Benefits offered by this job

Talent development

Hybrid work

Long Service awards

Annual health check-ups

Job summary

Persistent is seeking a Senior Databricks Data Engineer to design, develop, and optimize scalable data pipelines across all Persistent locations. The role emphasizes Databricks, PySpark, Python, and SQL expertise to enable AI-powered analytics and intelligent data access.

You will work on modern Lakehouse architectures, Delta Lake, Unity Catalog, and Databricks AI features to deliver scalable, secure data solutions and governance across enterprise platforms.

Qualifications

• 9–11 years in Data Engineering, Data Platform Development, or Data Warehousing.
• Minimum 5+ years in Data Engineering with hands-on Databricks experience.
• Strong expertise in Databricks, PySpark, Python, and SQL.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks, PySpark, and Python.
• Build batch and real-time data processing frameworks.
• Develop robust ETL/ELT solutions for ingesting, transforming, and processing data.
• Implement data quality, validation, reconciliation, and monitoring frameworks.
• Develop reusable components, standards, and best practices for data engineering projects.
• Ensure scalability, reliability, and performance across enterprise data platforms.
• Design and implement modern Databricks Lakehouse architectures.
• Work with Delta Lake, Unity Catalog, Databricks Workflows, and Databricks SQL.
• Optimize Spark jobs and cluster performance for large-scale data workloads.
• Implement enterprise-grade security, governance, and access controls within the Databricks ecosystem.
• Develop conversational analytics and AI-driven data solutions using Databricks Genie.
• Enable business users to interact with enterprise data using natural language queries.

Skills

Databricks

PySpark

Python

SQL

AI-powered analytics

Lakehouse architecture

Performance tuning

Data governance

Tools

Delta Lake

Unity Catalog

Databricks Workflows

Databricks SQL

Job description

About Position:

We are seeking a highly skilled Senior Databricks Data Engineer with strong expertise in Databricks, PySpark, Python, SQL, and AI-powered analytics solutions. The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines, implementing modern Lakehouse architectures, and leveraging Databricks AI capabilities such as Genie to enable intelligent data access and business insights.

• Role: Senior Databricks Data Engineer
• Location: All Persistent Locations
• Experience: 9 to 11 Years
• Job Type: Full Time Employment

What You’ll Do:

• Design, develop, and maintain scalable data pipelines using Databricks, PySpark, and Python.
• Build and optimize batch and real-time data processing frameworks.
• Develop robust ETL/ELT solutions for ingesting, transforming, and processing structured and unstructured data.
• Implement data quality, validation, reconciliation, and monitoring frameworks.
• Develop reusable components, standards, and best practices for data engineering projects.
• Ensure scalability, reliability, and performance across enterprise data platforms.
• Design and implement modern Databricks Lakehouse architectures.
• Work with Delta Lake, Unity Catalog, Databricks Workflows, and Databricks SQL.
• Optimize Spark jobs and cluster performance for large-scale data workloads.
• Implement enterprise-grade security, governance, and access controls within the Databricks ecosystem.
• Support platform modernization and continuous optimization initiatives.
• Develop conversational analytics and AI-driven data solutions using Databricks Genie.
• Enable business users to interact with enterprise data using natural language queries.
• Integrate Databricks AI capabilities to enhance self-service analytics and business intelligence.
• Collaborate with business teams to identify and develop AI-driven use cases.
• Build AI-assisted reporting, analytics, and decision-support solutions.
• Support Generative AI and intelligent analytics initiatives.
• Design and implement efficient data models supporting analytics, reporting, and AI workloads.
• Develop performance-optimized data transformations and business logic.
• Write complex SQL queries and scalable data processing solutions.
• Support business intelligence and analytics teams in delivering actionable insights.
• Drive data governance, consistency, and data quality initiatives.

Expertise You’ll Bring:

• 9 to 11 years of experience in Data Engineering, Data Platform Development, or Data Warehousing.
• Minimum 5+ years of experience in Data Engineering.
• Minimum 3+ years of hands-on experience with Databricks.
• Strong hands-on expertise in Databricks, PySpark, Python, SQL.
• Proven experience building large-scale enterprise data pipelines and platforms.
• Strong knowledge of Databricks Lakehouse architecture and implementation best practices.
• Hands-on experience with Delta Lake, Unity Catalog, Databricks Workflows, Databricks SQL.
• Strong experience in ETL/ELT development and distributed data processing.
• Expertise in data modeling, performance optimization, and data engineering best practices.
• Experience implementing data quality, governance, and security controls.
• Exposure to Databricks AI capabilities including Databricks Genie, Conversational Analytics, AI-Powered Analytics.
• Strong understanding of structured, semi-structured, and unstructured data processing.
• Experience writing advanced SQL queries and performance-tuning large datasets.
• Experience working in Agile software development environments.
• Excellent analytical, troubleshooting, and communication skills.
• Strong stakeholder management and collaboration capabilities.

Benefits:

• Competitive salary and benefits package
• Culture focused on talent development with quarterly growth opportunities and company-sponsored higher education and certifications
• Opportunity to work with cutting-edge technologies
• Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
• Annual health check-ups
• Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents

Values-Driven, People-Centric & Inclusive Work Environment:

Persistent is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds.

• We support hybrid work and flexible hours to fit diverse lifestyles.
• Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.
• If you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment

"Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind."