Senior Databricks Data Engineer

Persistent Systems; Pune District; Hybrid; INR 2,000,000 - 3,500,000; Full time

Benefits offered by this job

Flexible work hours

Long Service awards

Healthcare coverage

Job summary

Persistent Systems seeks a Senior Databricks Data Engineer to design and optimize scalable data pipelines using Databricks, PySpark, and Python. You will implement ETL/ELT solutions, ensure data quality, and contribute to modern Lakehouse architectures across enterprise platforms.

Ideal candidates have 9–11 years in data engineering, with 3+ years in Databricks, Delta Lake, and related tooling. A strong background in AI-powered analytics and governance is a plus.

Qualifications

• 9–11 years of experience in Data Engineering or Data Warehousing.
• 3+ years hands-on with Databricks and Delta Lake.
• Strong PySpark, Python, SQL and data modeling skills.
• Experience building large-scale data pipelines and Lakehouse architectures.
• Experience with AI-powered analytics and Databricks Genie is a plus.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks, PySpark, and Python.
• Build and optimize batch and real-time data processing frameworks.
• Develop robust ETL/ELT solutions for ingesting, transforming, and processing structured and unstructured data.
• Implement data quality, validation, reconciliation, and monitoring frameworks.
• Develop reusable components, standards, and best practices for data engineering projects.
• Ensure scalability, reliability, and performance across enterprise data platforms.
• Design and implement modern Databricks Lakehouse architectures.
• Work with Delta Lake, Unity Catalog, Databricks Workflows, and Databricks SQL.
• Optimize Spark jobs and cluster performance for large-scale data workloads.
• Implement enterprise-grade security, governance, and access controls within the Databricks ecosystem.
• Support platform modernization and continuous optimization initiatives.
• Develop conversational analytics and AI-driven data solutions using Databricks Genie.
• Enable business users to interact with enterprise data using natural language queries.
• Integrate Databricks AI capabilities to enhance self-service analytics and business intelligence.
• Collaborate with business teams to identify and develop AI-driven use cases.
• Build AI-assisted reporting, analytics, and decision-support solutions.
• Support Generative AI and intelligent analytics initiatives.
• Design and implement efficient data models supporting analytics, reporting, and AI workloads.
• Develop performance-optimized data transformations and business logic.
• Write complex SQL queries and scalable data processing solutions.
• Support business intelligence and analytics teams in delivering actionable insights.
• Drive data governance, consistency, and data quality initiatives.

Skills

Databricks

PySpark

Python

SQL

Data modeling

Spark tuning

AI-powered analytics

Tools

Delta Lake

Unity Catalog

Databricks Workflows

Databricks SQL

Job description

We are seeking a highly skilled Senior Databricks Data Engineer with strong expertise in Databricks, PySpark, Python, SQL, and AI-powered analytics solutions. The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines, implementing modern Lakehouse architectures, and leveraging Databricks AI capabilities such as Genie to enable intelligent data access and business insights.

• Location: All Persistent Locations
• Experience: 9 to 11 Years
• Job Type: Full Time Employment

What You'll Do:

• Design, develop, and maintain scalable data pipelines using Databricks, PySpark, and Python.
• Build and optimize batch and real-time data processing frameworks.
• Develop robust ETL/ELT solutions for ingesting, transforming, and processing structured and unstructured data.
• Implement data quality, validation, reconciliation, and monitoring frameworks.
• Develop reusable components, standards, and best practices for data engineering projects.
• Ensure scalability, reliability, and performance across enterprise data platforms.
• Design and implement modern Databricks Lakehouse architectures.
• Work with Delta Lake, Unity Catalog, Databricks Workflows, and Databricks SQL.
• Optimize Spark jobs and cluster performance for large-scale data workloads.
• Implement enterprise-grade security, governance, and access controls within the Databricks ecosystem.
• Support platform modernization and continuous optimization initiatives.
• Develop conversational analytics and AI-driven data solutions using Databricks Genie.
• Enable business users to interact with enterprise data using natural language queries.
• Integrate Databricks AI capabilities to enhance self-service analytics and business intelligence.
• Collaborate with business teams to identify and develop AI-driven use cases.
• Build AI-assisted reporting, analytics, and decision-support solutions.
• Support Generative AI and intelligent analytics initiatives.
• Design and implement efficient data models supporting analytics, reporting, and AI workloads.
• Develop performance-optimized data transformations and business logic.
• Write complex SQL queries and scalable data processing solutions.
• Support business intelligence and analytics teams in delivering actionable insights.
• Drive data governance, consistency, and data quality initiatives.

Expertise You'll Bring:

• 9 to 11 years of experience in Data Engineering, Data Platform Development, or Data Warehousing.
• Minimum 5+ years of experience in Data Engineering.
• Minimum 3+ years of hands‑on experience with Databricks.
• Strong hands‑on expertise in Databricks, PySpark, Python, SQL.
• Proven experience building large-scale enterprise data pipelines and platforms.
• Strong knowledge of Databricks Lakehouse architecture and implementation best practices.
• Hands‑on experience with Delta Lake, Unity Catalog, Databricks Workflows, Databricks SQL.
• Strong experience in ETL/ELT development and distributed data processing.
• Expertise in data modeling, performance optimization, and data engineering best practices.
• Experience implementing data quality, governance, and security controls.
• Exposure to Databricks AI capabilities including Databricks Genie, Conversational Analytics, AI‑Powered Analytics.
• Strong understanding of structured, semi‑structured, and unstructured data processing.
• Experience writing advanced SQL queries and performance‑tuning large datasets.
• Experience working in Agile software development environments.
• Excellent analytical, troubleshooting, and communication skills.
• Strong stakeholder management and collaboration capabilities.
• Competitive salary and benefits package
• Culture focused on talent development with quarterly growth opportunities and company‑sponsored higher education and certifications
• Opportunity to work with cutting‑edge technologies
• Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
• Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents

Values‑Driven, People‑Centric & Inclusive Work Environment:

Persistent is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds.

• We support hybrid work and flexible hours to fit diverse lifestyles.
• Our office is accessibility‑friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.
• If you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment

Let's unleash your full potential at Persistent - persistent.com/careers

"Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind."