Principal Data Engineer- Databricks & AI platform

Francisco Partners; Bengaluru; On-site; INR 4,200,000 - 7,000,000; Full time

Benefits offered by this job

Comprehensive health benefits

Generous paid time off

Tuition reimbursement

Wellness program

ERGs and charitable matching

Job summary

GoTo Bangalore is seeking a Principal Data Engineer to own our Databricks data platform. You’ll be the go‑to expert on Unity Catalog, Delta Lake, and Spark, and you’ll set the technical direction for how our teams build, govern, and scale data pipelines.

This hands‑on role operates at architect level across the data platform and ML/AI infrastructure. You’ll design the next version of the lakehouse, drive standards, mentor engineers, and lead migrations from legacy systems to Databricks.

Qualifications

• 12+ years in data engineering with hands-on Databricks experience.
• Deep expertise in Unity Catalog, Delta Lake, and Spark at scale.
• Experience designing medallion/lakehouse architectures.
• Strong SQL and Python skills.
• Experience with at least one cloud platform and IaC tooling.
• Experience with CI/CD for data pipelines and Databricks Asset Bundles.
• Track record of leading technical decisions and mentoring engineers.
• Experience deploying AI agents or AI-assisted tooling.

Responsibilities

• Own the architecture and roadmap for the Databricks platform: Unity Catalog, Delta Lake, cluster configuration, and workflow orchestration.
• Lead the architecture of the data platform, pipelines, and AI-ready datasets to support GenAI and ML use cases.
• Design and evolve medallion/lakehouse architecture to scale with the business.
• Set standards for data quality, security, and access control across the platform.
• Tune Spark jobs and cluster configs for performance and cost.
• Build and maintain core ETL frameworks and reusable components.
• Lead migrations from legacy platforms onto Databricks, including data modeling and pipeline redesign.
• Embed IaC and CI/CD practices into pipelines and platform infra.
• Troubleshoot production issues and perform root cause analysis.
• Mentor senior engineers and act as the technical escalation point.
• Drive adoption of spec-driven development and agentic AI tooling.
• Stay current on Databricks releases and adopt new capabilities when useful.

Skills

Databricks in production

Unity Catalog

Delta Lake

Spark performance tuning

SQL

Python

Cloud platforms (AWS/Azure/GCP)

Infrastructure as code

CI/CD for data pipelines

Leadership & mentoring

AI agent tooling adoption

Cross-functional communication

Tools

Airflow

dbt

Kafka

Job description

Where you’ll work:

Bangalore, KA, IN

Engineering at GoTo

We’re trailblazers in remote work technology—building powerful, flexible solutions that empower everyone to live their best life, both at work and beyond. With us, you’ll have the opportunity to chart new paths and help redefine how the world works. For us, AI isn’t just a buzzword; it’s a tool we use to deliver real, practical value to our customers and teams. We focus on solving meaningful problems, not just adding features for the sake of using AI. Here, growth takes many forms: you can expand your skills, take on new challenges, lead initiatives, and explore creative ideas. Join a GoTo product team and play a key role in transforming the workplace for millions of users worldwide—your work will truly make a difference.

About the Role

We're looking for a Principal Data Engineer to own our Databricks data platform. You'll be the go-to expert on Unity Catalog, Delta Lake, and Spark, and you'll set the technical direction for how our teams build, govern, and scale data pipelines. This is a hands-on role: you'll write code, review designs, and fix hard problems.

We're looking for a Principal Data Engineer who operates at architect level across our data platform and ML/AI infrastructure. You'll own the technical vision and multi-year roadmap for our Databricks Lakehouse, set the architectural standards that govern how Data Engineering teams build and ship, and be an active contributor to our ML and AI platform.

You won't be maintaining someone else's architecture. You'll be designing the next version of it, and you'll have the scope to make it matter.

Your Day to Day

• Own the architecture and roadmap for our Databricks platform: Unity Catalog, Delta Lake, cluster configuration, and workflow orchestration.

• Lead the architecture of our data platform, pipelines, and AI-ready datasets, so they're built to support GenAI and ML use cases, not just BI.

• Design and evolve our medallion (Bronze/Silver/Gold) lakehouse architecture so it scales with the business.

• Set standards for data quality, security, and access control across the platform, and make sure teams actually follow them.

• Tune Spark jobs and cluster configs for performance and cost. Find the waste, right-size the clusters, kill the idle spend.

• Build and maintain the core ETL frameworks and reusable components other teams build on top of.

• Lead migrations from legacy platforms onto Databricks, including data modeling and pipeline redesign.

• Embed infrastructure-as-code and CI/CD practices into how we build, deploy, and run pipelines and platform infrastructure, so deployments stay consistent and repeatable.

• Troubleshoot production issues, do root cause analysis, and put fixes in place that prevent repeat incidents.

• Mentor senior engineers and act as the technical escalation point across the Data teams.

• You’ll set the standard for spec-driven development and agentic AI tooling, then drive adoption for pipeline generation, testing, documentation, and incident response across the Data teams.

• Stay current on Databricks releases and bring in new capabilities when they solve a real problem, not just because they’re new.

What We’re Looking For

• 12+ years in data engineering, with at least 3+ years working hands-on with Databricks in production.

• Deep expertise in Unity Catalog, Delta Lake, and Spark performance tuning at scale.

• Real experience designing and running medallion/lakehouse architectures.

• Strong SQL and Python.

• Experience with at least one major cloud platform (AWS, Azure, or GCP) and infrastructure-as-code tooling.

• Experience with CI/CD for data pipelines, including Databricks Asset Bundles.

• A track record of leading technical decisions and mentoring other engineers.

• Experience rolling out AI agents or AI‑assisted tooling across an engineering team. You've set standards and gotten a team to adopt them.

• Clear communicator who can work across engineering, analytics, and business stakeholders.

Nice to Have

• Experience with orchestration tools like Airflow or dbt.

• Experience with streaming/messaging systems like Kafka.

• Exposure to GenAI, RAG, or LLM‑based data applications.

• Experience leading a legacy data warehouse to Databricks migration.

What We Offer

At GoTo, we care about helping our people succeed at work and feel supported in life. Our employee benefits and programs are designed to support your well‑being, growth, and sense of belonging. Here's what you can expect as part of our team:

• Comprehensive health benefits
• Generous paid time off, including paid holidays, volunteer days, quarterly self‑care days, and company‑designated no‑meeting days
• Tuition reimbursement and access to instructor‑led and on‑demand learning and development programs
• The Thrive Global Wellness Program, a confidential Employee Assistance Program (EAP), a wellness app and one‑on‑one wellness coaching
• Employee‑led communities and programs, including Employee Resource Groups (ERGs), GoTo Gives, and charitable matching

We work hard to create an environment where everyone feels welcome, respected, and able to contribute. Building a culture of belonging isn't just something we talk about - it's part of how we work every day.

Specific benefits and offerings may vary by country in line with local regulations and market practices.

At GoTo, you’ll find the flexibility, resources, and support you need to thrive—at work, at home, and everywhere in between. You’ll work towards a shared goal with an open‑minded, cohesive team that’s greater than the sum of its parts. We’re committed to creating an inclusive space for everyone, because we know unique perspectives make us a stronger company and community. Join us and be part of a company that invests in your future, where together we’ll Be Real, Think Big, Move Fast, Keep Growing, and stay Customer Obsessed.