EY - GDS Consulting - AI and DATA - Kafka Engineer - Manager

EY; Bengaluru; Presencial; INR 3.000.000 - 4.200.000; Jornada completa

Descripción de la vacante

EY in Bengaluru is seeking a Manager - Data Engineering & Data Architecture to design and build modern data Lakehouse platforms using Databricks on AWS and Kafka streaming technologies. The role focuses on delivering scalable data pipelines, governance, and enterprise analytics-ready data products.

The ideal candidate will lead data platform initiatives, drive engineering best practices, and collaborate with stakeholders to enable BI, analytics, and AI use cases in a global delivery environment.

Formación

• Hands‑on Databricks, Spark and Kafka experience.
• Experience designing modern Data Lakehouse architectures.
• Strong data modelling and ETL/ELT design skills.
• Knowledge of cloud platforms (AWS) and data platforms architecture.
• Experience leading technical delivery and mentoring teams.

Responsabilidades

• Design and implement modern Data Lakehouse platforms on AWS with Databricks.
• Build scalable ETL/ELT pipelines and streaming solutions with Kafka.
• Develop end‑to‑end data models (Raw/Silver/Gold) for enterprise reporting.
• Establish data platform standards, governance, and CI/CD practices.
• Provide technical leadership and mentorship to engineering teams.

Conocimientos

Databricks

Apache Spark

Kafka Streaming

Semi-structured data

ETL/ELT

Data Lakehouse

Data Modeling

Data Warehousing

SQL

Real-Time Processing

GitHub

CI/CD

Agile Delivery

Educación

Bachelor’s/Master’s in CS/IT/Engineering

Herramientas

Azure DevOps

Databricks Asset Bundles (DAB)

GitHub

Descripción del empleo

EY - GDS Consulting - AI and DATA - Kafka Engineer - Manager

Other locations: Primary Location Only

Date: Oct 12, 2026

Requisition ID: 1751110

At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all.

EY Hiring Requisition

Manager - Data Engineering & Data Architecture

Experience: 10-12 Years

Job Summary

EY is seeking a highly skilled Data Engineering and Data Architecture professional with strong expertise in Databricks, Kafka, AWS, and modern Data Lakehouse architecture.

The ideal candidate will have hands‑on experience designing and implementing scalable data platforms, building enterprise‑grade data pipelines, and delivering analytics‑ready data products.

The candidate should possess deep technical knowledge across the data engineering lifecycle, from ingestion and transformation to modelling and consumption, while driving delivery through engineering best practices, automation, and stakeholder collaboration.

Key Responsibilities

• Design and implement modern Data Lakehouse platforms using Databricks on AWS
• Build ETL/ELT pipelines
• Implement Kafka streaming solutions
• Design Fact and Dimension models
• Implement CI/CD using Azure DevOps and GitHub
• Provide technical leadership

Data Architecture & Platform Engineering

• Design and implement modern cloud-based Data Lakehouse platforms using Databricks on AWS.
• Define and implement scalable Raw, Silver, and Gold layer architectures.
• Architect end-to-end data solutions supporting reporting, BI, advanced analytics, and AI use cases.
• Establish data platform standards, governance controls, and engineering best practices.
• Design, develop, and optimize large-scale ETL/ELT pipelines.
• Build high-performance Spark-based data processing solutions.
• Develop reusable and metadata-driven ingestion and transformation frameworks.
• Ensure data quality, reliability, scalability, and operational efficiency across data pipelines.
• Implement data validation, reconciliation, and monitoring mechanisms.
• Handling semi‑structured data formats- json, xml

Streaming & Real‑Time Data Processing

• Design and implement real‑time data ingestion frameworks using Kafka.
• Build scalable streaming solutions integrating Kafka with Databricks and AWS services.
• Optimize streaming workloads for performance and resilience.

Data Modeling

• Design conceptual, logical, and physical data models.
• Create dimensional models, Fact tables, and Dimension tables for enterprise reporting and analytics.
• Implement data models aligned with business and analytical requirements.
• Leverage Azure DevOps, GitHub, Databricks Asset Bundles (DAB), and automation frameworks.
• Promote version control, code reviews, testing, and release management practices.

Stakeholder & Team Management

• Collaborate with business stakeholders, architects, and delivery teams to translate requirements into scalable solutions.
• Provide technical leadership and mentoring to engineering teams.
• Contribute to solution design workshops, estimations, and architecture reviews.

Required Skills

Core Technical Skills

• Databricks (Expert Level or Databricks Champion)
• Apache Spark (PySpark / Spark SQL)
• Kafka Streaming
• Semi-structured data processing like json, xmls
• ETL/ELT Design and Development
• Data Lakehouse Architecture
• Data Modeling and Dimensional Modeling
• Data Warehousing Concepts
• SQL and Performance Optimization
• Real‑Time Data Processing
• GitHub
• CI/CD Implementation
• Agile Delivery Methodology

Other relevant skills

• Strong problem‑solving and analytical skills.
• Excellent communication and stakeholder management capabilities.
• Ability to work in a global delivery environment.
• Strong ownership mindset with focus on quality and client value.
• Demonstrated leadership and mentoring capabilities.

Preferred Experience

• Hands‑on implementation of Raw, Silver, and Gold architectural patterns.
• Experience building metadata‑driven ingestion and transformation frameworks.
• Exposure to Insurance, Banking, or Financial Services domain data platforms.
• Experience supporting analytics, reporting, and AI/ML workloads on Databricks.
• Experience leading distributed teams and managing technical delivery.

Certifications (Preferred)

One or more of the following certifications are highly desirable:

• Databricks Certified Professional Certifications
• Databricks Lakeflow Certification

Educational Qualification

Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related discipline.

EY | Building a better working world

EY exists to build a better working world, helping to create long‑term value for clients, people and society and build trust in the capital markets.

Enabled by data and technology, diverse EY teams in over 150 countries provide trust through assurance and help clients grow, transform and operate.

Working across assurance, consulting, law, strategy, tax and transactions, EY teams ask better questions to find new answers for the complex issues facing our world today.