Lead Databricks Data Engineer

Persistent Systems Limited; Bengaluru; Híbrido; INR 3 500 000 - 6 000 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Hybrid work model

Medical insurance

Learning @ Work

Flexible hours

Long Service awards

Resumo da oferta

Persistent Systems Limited is seeking a Lead Databricks Data Engineer to design and build scalable data pipelines and analytics platforms. You will work across Azure, AWS, or Google Cloud, delivering enterprise-grade data solutions and streaming capabilities.

The role emphasizes hands-on Databricks, PySpark, SQL, and Delta Lake with collaboration across architects, data scientists, and business stakeholders. Hybrid work model in Bengaluru, India is supported.

Qualificações

• 8–12 years of Data Engineering experience.
• Strong hands-on Databricks Lakehouse Platform expertise.
• Proficiency in PySpark, Spark SQL, Python, and SQL.
• Experience designing scalable ETL/ELT pipelines and data integration frameworks.
• Deep knowledge of Delta Lake, data warehousing concepts, and query optimization.
• Experience with Azure, AWS, or Google Cloud environments.

Responsabilidades

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks.
• Build and optimize large-scale data processing with PySpark and Spark SQL.
• Develop ingestion frameworks for structured, semi-structured, and unstructured data.
• Implement batch and real-time processing on Databricks Lakehouse Platform.
• Support Delta Lake implementations and Lakehouse architectures.
• Collaborate with Architects, Data Scientists, and Product Owners to deliver data solutions.
• Adopt CI/CD for data engineering workloads and provide production support.

Conhecimentos

Databricks Lakehouse

PySpark

Spark SQL

Python

SQL

ETL/ELT

Cloud platforms

CI/CD

Kafka/Streaming

Git / DevOps

Data Warehousing

Ferramentas

Databricks

Azure Data Factory

AWS Glue

Azure DevOps

Git

Unity Catalog

Descrição da oferta de emprego

We are an AI-led, platform-driven Digital Engineering and Enterprise Modernization partner, combining deep technical expertise and industry experience to help our clients anticipate what’s next. Our offerings and proven solutions create a unique competitive advantage for our clients by giving them the power to see beyond and rise above. We work with many industry-leading organizations across the world, including 20 Fortune 50 companies and 4 of the 5 top banks in both the US and India, and numerous innovators across the healthcare ecosystem.

We are looking for a skilled Lead Databricks Data Engineer with strong expertise in Data Engineering, Big Data technologies, and the Databricks Lakehouse Platform. The ideal candidate will be responsible for designing, developing, and supporting scalable data pipelines, enterprise analytics solutions, and cloud-native data platforms. The role requires deep hands-on experience in Databricks, PySpark, SQL, ETL/ELT frameworks, and cloud technologies. You will work closely with architects, data scientists, business stakeholders, and engineering teams to deliver high-performance, scalable, and reliable data solutions that enable enterprise-wide analytics and data-driven decision-making.

• Role: Lead Databricks Data Engineer
• Experience: 8 to 12 Years
• Job Type: Full Time Employment

What You'll Do:

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks.
• Build and optimize large-scale data processing workflows using PySpark and Spark SQL.
• Develop data ingestion frameworks for structured, semi-structured, and unstructured data sources.
• Implement batch and real-time data processing solutions on the Databricks Lakehouse Platform.
• Design and support Delta Lake implementations and Lakehouse architectures.
• Optimize Databricks workloads for performance, scalability, reliability, and cost efficiency.
• Develop data transformation, validation, cleansing, and reconciliation frameworks.
• Support enterprise data modeling and data warehousing initiatives.
• Implement monitoring, alerting, and troubleshooting mechanisms for data pipelines.
• Integrate Databricks with cloud services, APIs, enterprise applications, and third-party systems.
• Develop cloud-native data solutions across Azure, AWS, or Google Cloud environments.
• Support CI/CD implementation and deployment automation for data engineering workloads.
• Collaborate with Architects, Business Analysts, Data Scientists, and Product Owners to deliver business-driven data solutions.
• Participate in solution design discussions, architecture reviews, and code review processes.
• Support production deployments, incident resolution, and performance tuning activities.
• Create technical documentation, design specifications, and operational runbooks.
• Mentor junior engineers and promote data engineering best practices across teams.

Expertise You'll Bring:

• 8–12 years of experience in Data Engineering, Big Data, and Analytics platform development.
• Strong hands-on expertise with Databricks Lakehouse Platform and enterprise-scale data engineering solutions.
• Advanced proficiency in PySpark, Spark SQL, Python, and SQL.
• Extensive experience designing and developing scalable ETL/ELT pipelines and data integration frameworks.
• Deep understanding of Delta Lake architecture, transaction management, and performance optimization techniques.
• Strong knowledge of Data Warehousing concepts, dimensional modeling, and enterprise analytics platforms.
• Experience working with structured, semi-structured, and unstructured data at scale.
• Hands-on experience building batch and real-time data processing solutions.
• Strong expertise in data ingestion, transformation, validation, cleansing, and reconciliation processes.
• Experience integrating enterprise applications, APIs, cloud services, and diverse data sources.
• Familiarity with Azure Data Factory, AWS Glue, or equivalent cloud-native data integration services.
• Experience working with Azure, AWS, or Google Cloud Platform environments.
• Strong understanding of Git, source control practices, and collaborative development methodologies.
• Experience implementing and managing CI/CD pipelines for data engineering workloads.
• Knowledge of Kafka, Event Hub, or similar streaming technologies for real-time data processing.
• Familiarity with Unity Catalog, metadata management, and enterprise data governance frameworks.
• Strong understanding of schema design, data modeling, and query optimization techniques.
• Experience using Azure DevOps and deployment automation frameworks.
• Exposure to AI/ML data pipelines, feature engineering, and model data preparation workflows.
• Strong troubleshooting, production support, and root cause analysis capabilities.
• Experience optimizing Databricks workloads for enterprise-scale performance and reliability.
• Ability to independently lead medium-to-large data engineering initiatives.
• Experience mentoring team members and conducting technical reviews.
• Excellent analytical, communication, and stakeholder management skills.
• Strong commitment to code quality, testing, engineering best practices, and continuous improvement.
• Ability to contribute to architecture discussions and strategic data platform initiatives.
• Competitive salary and benefits package
• Culture focused on talent development with quarterly growth opportunities and company-sponsored higher education and certifications
• Opportunity to work with cutting-edge technologies
• Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
• Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents

Values-Driven, People-Centric & Inclusive Work Environment:

Persistent is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds.

• We support hybrid work and flexible hours to fit diverse lifestyles.
• Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.
• If you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment

“Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind.”