Lead - Data Engineer

Bridgestone - Global Capability Center; Bengaluru; On-site; INR 4,000,000 - 6,400,000; Full time

Job summary

Bridgestone - Global Capability Center in Bengaluru is seeking a Lead Data Engineer for its Enterprise Data Platform, a hands-on leader responsible for building and operating ingestion and curation pipelines across multi-cloud environments using Databricks as the core platform.

This role will mentor a team of data engineers, set standards, and partner with Platform Engineering, Solution Engineers, and business units to deliver reliable, scalable data solutions for analytics and AI initiatives.

Qualifications

• Bachelor's degree in CS or equivalent.
• 8+ years data engineering/IT dev, incl. 5+ cloud data eng and 3+ guiding engineers.
• Deep Databricks experience: Spark, Delta Lake, Unity Catalog, workflows.
• Hands-on Genie experience: building Genie Spaces with tables, queries, metrics.
• Strong data modeling, medallion architecture, partitioning, CDC, schema evolution.
• Proven enterprise-scale pipeline development across clouds (AWS/Azure).
• Expert Python and SQL.
• Experience with orchestration at scale (Databricks Workflows, Step Functions, ADF).
• Knowledge of data governance, lineage, quality, access controls.
• CI/CD with Git/Azure DevOps or equivalent.

Responsibilities

• Designs and operates data ingestion/transformation pipelines across clouds.
• Mentors data engineers; conducts design/code reviews for standards.
• Builds pipelines across AWS and Azure; guides migration to Databricks.
• Defines guardrails and reusable patterns for self-service data products.
• Partners with BI/analytics to deliver curated data products and queries.
• Maintains Genie Spaces with clear metrics and trustworthy answers.
• Sets AI tooling strategy within the team and production policy.
• Drives data quality, lineage, observability, and access control.
• Participates in architectural reviews and governance discussions.
• Continuous improvement: CI/CD, IaC, testing, automation.
• Monitors pipelines; leads RCAs and SOPs to stabilize ops.
• Coordinates with Databricks/AWS/Microsoft for issue resolution.
• Other duties as assigned.

Skills

Mentoring/Data engineering leadership

Communication skills

Agile delivery

End-to-end testing

Education

Bachelor's degree in Computer Science or equivalent

Tools

Databricks

PySpark

SQL

Delta Lake

Unity Catalog

AWS Glue

Azure Data Factory

Databricks Workflows

Step Functions

Redshift

S3

ADLS

Genie Spaces

Job description

Position Summary

The Lead Data Engineer, Enterprise Data Platform (EDP) is a hands‑on technical leader responsible for building and operating the ingestion and curation layers of Bridgestone's multi‑cloud Enterprise Data Platform. The EDP is a platform‑as‑a‑service: business units build their own data products on top of it, within guardrails and governance standards this role helps define and enforce. The platform runs on a Databricks and Unity Catalog backbone spanning AWS and Azure, delivering curated data to business units, BI and analytics teams, and conversational query experiences.

This role writes code, designs pipelines, and sets the technical standard for a team of Data Engineers. This role partners closely with Platform Engineering, Solution Engineers, Product Owners, and business unit engineering teams. Deep Databricks expertise, strong data engineering fundamentals, and the ability to operate credibly in a large, matrixed enterprise are essential.

Job Duties

• Designs, builds, and operates industrialized data ingestion and transformation pipelines from source systems through the silver layer of a medallion architecture, using Databricks (Spark, Unity Catalog, workflows) as the core platform.
• Serves as the team's Databricks subject matter expert data modeling in Delta, job and cluster design, performance and cost tuning, and Unity Catalog governance patterns.
• Guides, coaches, and technically mentors other Data Engineers; conducts design and code reviews to ensure architecture, platform, and solution standards are followed.
• Builds and maintains pipelines across multiple clouds — primarily AWS (Glue, Step Functions, Redshift, Aurora, Transfer Family, S3) and Azure (Data Factory, ADLS) — and plans the migration of native cloud pipelines to Databricks as a destination technology.
• Defines and enforces the guardrails, standards, and reusable patterns that business units follow when building their own projects on the platform, balancing self‑service velocity against governance requirements.
• Partners with the BI and analytics teams to deliver curated, well‑modeled, well‑documented data products fit for reporting and for conversational query experiences such as Databricks Genie Space.
• Builds and maintains Databricks Genie Spaces over curated data, including the table selection, instructions, sample queries, and metric definitions that make natural‑language answers trustworthy. Reviews what Genie returns, and when an answer is wrong, fixes the underlying data model rather than patching the question.
• Sets the team's working practice for AI‑assisted engineering. Decides where AI tooling belongs in pipeline development, code review, testing, and documentation, where it does not, and what review is required before AI‑assisted work reaches production.
• Drives data quality, lineage, observability, and access control practices in partnership with data governance and security teams.
• Participates in early design reviews and architectural strategy for data modeling, design, and implementation within the responsible data domains.
• Drives continuous improvement and automation — CI/CD, infrastructure as code, testing, and operational tooling — and solicits recommendations from the team to ensure the right solutions are implemented at the right time.
• Monitors production pipelines, leads triage and root cause analysis for data incidents, and documents RCAs and SOPs to standardize and stabilize daily operations.
• Coordinates with vendors and platform partners (Databricks, AWS, Microsoft) to resolve product defects and escalations.
• Other duties as assigned.

Required Qualifications

• Bachelor's degree in computer science, computer engineering, information systems, or equivalent work experience.
• Minimum of 8 years in data engineering or IT development, including 5 years of cloud data engineering and 3 years leading or technically guiding other engineers.
• Deep, hands‑on Databricks experience — Spark (PySpark/SQL), Delta Lake, Unity Catalog, Databricks workflows, and job/cluster performance and cost optimization. This is the core technology of the platform.
• Hands‑on working experience with Databricks Genie, including building Genie Spaces from scratch: curating the underlying tables, writing space instructions and sample queries, defining metrics, and tuning a space until its answers hold up under real business questions.
• Strong data engineering fundamentals: data modeling, dimensional and medallion architecture patterns, partitioning, incremental and CDC ingestion, idempotency, schema evolution, and backfill strategy.
• Demonstrated hands‑on pipeline development at enterprise scale — not solely oversight or coordination. This role builds.
• Production experience across multiple clouds, primarily AWS and Azure, including managed data services, storage, networking basics, and identity and access management.
• Expert‑level Python and SQL.
• Experience with orchestration at scale (Databricks Workflows, Step Functions, Azure Data Factory, or equivalent) for complex, dependency‑heavy pipelines.
• Working knowledge of data governance concepts, tools, and processes in a complex organizational environment — cataloging, lineage, data quality, and fine‑grained access control.
• Experience with Git and CI/CD (Azure DevOps or equivalent) to promote code and release packages through environments.
• Experience working in large enterprise environments: matrixed teams, multiple business units with differing data maturity, change management, and formal release processes.
• Ability to perform end‑to‑end testing and debug issues across distributed cloud services.
• Excellent communication skills — able to explain technical constraints and trade‑offs to engineers, business stakeholders, and leadership.
• Practical experience with Agile delivery practices.
• Uses AI tooling as part of a normal working day rather than an occasional experiment. Candidates will be asked to walk through their own workflow in the interview: which tools, at what points in the work, what they verify before trusting the output, and what actually got faster as a result.