Lead Databricks Data Engineer

Persistent Systems; Pune District; Híbrido; INR 4 000 000 - 6 600 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Competitive salary and benefits

Talent development with certifications

Flexible work hours

Long Service awards

Insurance coverage: group term life, P

Resumo da oferta

Persistent Systems is seeking a Lead Databricks Data Engineer to design and optimize enterprise-scale data platforms and Lakehouse architectures. You will lead pipelines using Databricks, PySpark, Python and SQL, and drive AI-powered analytics with Genie.

Expect collaboration with architects, data scientists, and product owners to deliver scalable, secure data solutions. The role requires 12–15 years of experience, strong leadership, and a hands-on approach to cloud-native architectures, data

Qualificações

• 12–15 years of data engineering experience with enterprise-scale platforms.
• Deep hands-on with Databricks, PySpark, Python, and SQL.
• Experience designing Lakehouse architectures and modern data platforms.
• Proven ability to lead data engineering teams and governance programs.
• Strong stakeholder management, communication, and Agile delivery experience.

Responsabilidades

• Design, develop, and optimize scalable data pipelines in Databricks.
• Lead modern Lakehouse architectures and AI-powered analytics initiatives.
• Implement data quality, governance, security, and lineage controls.
• Mentor engineers and drive best practices, reviews, and architecture decisions.
• Collaborate with Data Scientists, Architects, and product owners to deliver AI-driven solutions.

Conhecimentos

Databricks

PySpark

Python

SQL

AI analytics

Leadership

Agile

Stakeholder mgmt

Ferramentas

Delta Lake

Unity Catalog

Databricks SQL

Databricks Workflows

AWS S3

AWS Glue

AWS Lambda

Amazon EMR

Amazon Redshift

AWS IAM

CloudWatch

Descrição da oferta de emprego

We are seeking a highly experienced Lead Databricks Data Engineer with strong expertise in Databricks, PySpark, Python, SQL, and AI-powered analytics solutions. The ideal candidate will lead the design, development, and optimization of enterprise-scale data platforms, modern Lakehouse architectures, and AI-driven analytics solutions. This role requires hands-on experience in building scalable data pipelines, implementing cloud-native data architectures, leveraging Databricks AI capabilities such as Genie, and driving enterprise data modernization initiatives. The successful candidate will work closely with Data Architects, Data Scientists, Product Owners, and business stakeholders to deliver innovative data solutions that enable intelligent decision-making and business growth.

• Role: Lead Databricks Data Engineer
• Location: All Persistent Locations
• Experience: 12 to 15 Years
• Job Type: Full Time Employment

What You'll Do:

• Design, develop, and maintain scalable enterprise-grade data pipelines using Databricks, PySpark, Python, and SQL.
• Build and optimize batch and real-time data processing frameworks for high-volume data workloads.
• Develop robust ETL/ELT solutions for structured, semi-structured, and unstructured data.
• Implement data quality, validation, reconciliation, and monitoring frameworks.
• Establish reusable components, standards, and best practices for enterprise data engineering.
• Lead initiatives focused on scalability, reliability, and performance optimization of data platforms.
• Design and implement modern Databricks Lakehouse architectures.
• Work extensively with Delta Lake, Unity Catalog, Databricks Workflows, and Databricks SQL.
• Tune Spark jobs and optimize cluster configurations for large-scale processing workloads.
• Implement enterprise-grade security, governance, lineage, and access management controls.
• Define architectural standards and best practices for Databricks implementations.
• Develop AI-driven analytics and conversational data solutions using Databricks Genie.
• Enable self-service analytics through natural language interactions with enterprise data.
• Design and implement AI-powered reporting and business intelligence solutions.
• Collaborate with business users to identify and deliver AI-driven analytics use cases.
• Integrate Databricks AI capabilities to improve decision-making and operational efficiency.
• Support Generative AI, intelligent analytics, and advanced data access initiatives.
• Integrate data from enterprise applications, databases, APIs, cloud platforms, and third-party systems.
• Work extensively with AWS services including Amazon S3, AWS Glue, AWS Lambda, Amazon EMR, Amazon Redshift, AWS IAM, AWS CloudWatch.
• Build scalable cloud-native data architectures supporting modern analytics workloads.
• Provide technical leadership to data engineering teams.
• Mentor and coach junior engineers and promote engineering excellence.
• Collaborate with Data Architects, Data Scientists, Product Owners, and business stakeholders.
• Participate in architecture reviews, design discussions, and strategic planning initiatives.
• Lead code reviews and ensure adherence to coding standards and best practices.
• Support troubleshooting, performance optimization, and production issue resolution.
• Actively contribute to innovation initiatives involving AI-powered analytics and modern data platforms.

Expertise You'll Bring:

• 12 to 15 years of experience in Data Engineering, Data Platform Engineering, or Data Warehousing.
• Strong hands-on expertise in Databricks platform implementation and administration.
• Advanced experience with Databricks, PySpark, Python, SQL.
• Proven experience designing and delivering enterprise-scale data engineering solutions.
• Expertise in Databricks Lakehouse architecture and modernization programs.
• Strong experience with Delta Lake, Unity Catalog, Databricks SQL, Databricks Workflows.
• Experience building high-performance ETL/ELT frameworks and distributed data processing pipelines.
• Deep understanding of data modeling, data architecture, and enterprise data management principles.
• Hands-on experience with Databricks Genie and AI-powered analytics capabilities.
• Experience implementing conversational analytics and intelligent data access solutions.
• Strong expertise in performance tuning, query optimization, and Spark workload optimization.
• Experience implementing data governance, security, lineage, and compliance frameworks.
• Strong stakeholder management and leadership capabilities.
• Experience working in Agile delivery environments.
• Excellent communication, presentation, and problem-solving skills.
• Competitive salary and benefits package
• Culture focused on talent development with quarterly growth opportunities and company-sponsored higher education and certifications
• Opportunity to work with cutting-edge technologies
• Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
• Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents

Values-Driven, People-Centric & Inclusive Work Environment:

Persistent is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds.

• We support hybrid work and flexible hours to fit diverse lifestyles.
• Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.
• If you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment

Let's unleash your full potential at Persistent - persistent.com/careers

"Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind."