Lead Consultant- Databricks Developer !

Genpact; Pune District; On-site; INR 10,00,000 - 15,00,000; Full time

Job summary

A global technology solutions provider is seeking a Lead Consultant - Databricks Developer in Pune. The role involves designing scalable data pipelines, optimizing workflows, and mentoring junior developers. The ideal candidate has significant experience with Databricks and cloud technologies, and a strong background in data engineering. This position offers an opportunity to solve real-world problems in a fast-paced environment.

Qualifications

• Experience in data engineering with at least Databricks experience.
• End-to-end implementation of at least 2 Databricks projects.
• Strong background in batch and streaming data pipelines.

Responsibilities

• Design and develop scalable data pipelines using Databricks.
• Implement ETL/ELT frameworks leveraging Lakehouse architecture.
• Optimize data models, queries, and workflows.

Skills

Databricks experience

Data engineering

Python

SQL & Spark-SQL

Performance optimization

Cloud expertise (Azure/AWS)

Team collaboration

Problem-solving

Tools

Apache Spark

Databricks

Hive

CI/CD tools

Job description

Job Description - Lead Consultant- Databricks Developer

Ready to shape the future of work?

At Genpact, we don’t just adapt to change—we drive it. AI and digital innovation are redefining industries, and we’re leading the charge. Genpact’s AI Gigafactory, our industry-first accelerator, is an example of how we’re scaling advanced technology solutions to help global enterprises work smarter, grow faster, and transform at scale.

If you thrive in a fast-moving, tech-driven environment, love solving real-world problems, and want to be part of a team that’s shaping the future, this is your moment.

Genpact (NYSE: G) is an advanced technology services and solutions company that delivers lasting value for leading enterprises globally. Through our deep business knowledge, operational excellence, and cutting-edge solutions – we help companies across industries get ahead and stay ahead.

Inviting applications for the role of Lead Consultant- Databricks Developer

In this role, the Databricks Developer is responsible for solving real-world cutting-edge problems to meet both functional and non-functional requirements.

Responsibilities

• Design and develop scalable data pipelines using Databricks (PySpark/SQL/Delta Live Tables).
• Implement ETL/ELT frameworks leveraging the Lakehouse (Bronze, Silver, Gold) architecture.
• Implement and manage data governance using Unity Catalog to ensure secure access, compliance, and centralized management of data, users, and permissions across the Databricks Lakehouse.
• Optimize data models, queries, and workflows for scalability, cost, and performance.
• Act as Databricks SME and provide guidance on best practices, governance, and security.
• Mentor junior developers, review code, and enforce coding standards.
• Integrate Databricks with cloud storage, APIs, warehouses, and BI tools.
• Implement orchestration using ADF, Airflow, Step Functions, or Databricks Workflows.
• Build reusable accelerators and frameworks for ingestion, transformation, and monitoring.
• Enable data quality, validation, and reconciliation (using tools like Great Expectations or custom).
• Set up monitoring, logging, and alerting dashboards for pipeline health.
• Collaborate with business stakeholders, architects, and analysts to deliver solutions.
• Support migration of legacy pipelines into Databricks Lakehouse.
• Contribute to architectural decisions, POCs, and innovation initiatives.

Qualifications we seek in you!

• Experience in data engineering with at least Databricks experience.
• End-to-end implementation of at least 2 Databricks projects (migration/integration).
• Strong background in batch and streaming data pipelines.
• Proficiency in Python (preferred) or Scala for Spark-based development.
• Expertise in SQL & Spark-SQL, data structures, and algorithms.
• Deep knowledge of Databricks components: Delta Lake, DLT, dbConnect, REST API 2.0, Workflows orchestration.
• Strong in performance optimization for pipelines (efficiency, scalability, cost reduction).
• Hands-on experience with Apache Spark, Hive, and Lakehouse architecture.
• Cloud expertise (Azure/AWS) includes storage (ADLS/S3), messaging (ASB/SQS), compute (ADF/Lambda), and databases (CosmosDB/DynamoDB/Cloud SQL).
• Experience writing unit tests and integration tests for data pipelines.
• Ability to work with architects and lead engineers to design solutions meeting functional & non-functional requirements.
• Team player with experience in teams of 5+ engineers.
• Strong communication and client-facing skills.
• Keeps updated with emerging technologies and industry trends.
• Strong analytical and problem-solving abilities.
• Positive attitude towards continuous learning and upskilling.

Good to have: Databricks SQL Endpoint understanding, LakeflowConnect, Lakeflow Declarative Pipelines, CI/CD experience to build the pipeline for Databricks jobs, migration project to build Unified data platform, knowledge of DBT, docker and Kubernetes, Certification on Databricks Associate level, Any one Cloud Certification (AWS/Azure) Practitioner or Associate Level.

Genpact is an Equal Opportunity Employer and considers applicants for all positions without regard to race, colour, religion or belief, sex, age, national origin, citizenship status, marital status, military/veteran status, genetic information, sexual orientation, gender identity, physical or mental disability or any other characteristic protected by applicable laws.