Principal Data Engineer- Databricks & AI platform

306 - GoTo Technologies India Private Limited; Bengaluru; On-site; INR 4,000,000 - 6,000,000; Full time

Benefits offered by this job

Comprehensive health benefits

Generous paid time off & holidays

Tuition reimbursement & learning

Job summary

GoTo Technologies India Private Limited is seeking a Principal Data Engineer to own and evolve the Databricks data platform, including Unity Catalog, Delta Lake, and Spark performance tuning. You will set architectural standards, oversee multi-year roadmaps, and mentor senior engineers across Data teams.

You will drive data pipelines, medallion lakehouse design, and AI-enabled tooling adoption while collaborating with analytics, engineering, and business stakeholders.

Qualifications

• 12+ years in data engineering with hands-on Databricks in production.
• Deep expertise in Unity Catalog, Delta Lake, and Spark at scale.
• Experience designing medallion/lakehouse architectures.
• Strong SQL and Python.
• Experience with AWS, Azure, or GCP and infrastructure-as-code.
• Experience with CI/CD for data pipelines, including Databricks Asset Bundles.
• Track record of leading technical decisions and mentoring engineers.
• Experience rolling out AI agents or AI-assisted tooling.

Responsibilities

• Own architecture and roadmap for Databricks platform (Unity Catalog, Delta Lake, cluster config).
• Lead data platform and AI-ready dataset design to support GenAI and ML use cases.
• Design medallion lakehouse architectures scalable with business needs.
• Set data quality, security, and access standards across platforms.
• Tune Spark jobs and cluster configs for performance and cost.
• Migrate legacy data warehouses to Databricks; redesign pipelines.

Skills

Databricks

Unity Catalog

Delta Lake

Spark

SQL

Python

Cloud platforms

CI/CD for data pipelines

Mentoring

Architecture & roadmap

Tools

Databricks

Job description

Where you’ll work: Bangalore, KA, IN Engineering at GoTo We’re trailblazers in remote work technology - building powerful, flexible solutions that empower everyone to live their best life, both at work and beyond. With us, you’ll have the opportunity to chart new paths and help redefine how the world works. For us, AI isn’t just a buzzword; it’s a tool we use to deliver real, practical value to our customers and teams. We focus on solving meaningful problems, not just adding features for the sake of using AI. Here, growth takes many forms: you can expand your skills, take on new challenges, lead initiatives, and explore creative ideas. Join a GoTo product team and play a key role in transforming the workplace for millions of users worldwide - your work will truly make a difference.

About the Role

We’re looking for a Principal Data Engineer to own our Databricks data platform. You'll be the go‑to expert on Unity Catalog, Delta Lake, and Spark, and you'll set the technical direction for how our teams build, govern, and scale data pipelines.

We’re looking for a Principal Data Engineer who operates at architect level across our data platform and ML/AI infrastructure. You'll own the technical vision and multi-year roadmap for our Databricks Lakehouse, set the architectural standards that govern how Data Engineering teams build and ship, and be an active contributor to our ML and AI platform. You won’t be maintaining someone else’s architecture. You’ll be designing the next version of it, and you’ll have the scope to make it matter.

Our Day to Day

• Own the architecture and roadmap for our Databricks platform: Unity Catalog, Delta Lake, cluster configuration, and workflow orchestration.
• Lead the architecture of our data platform, pipelines, and AI‑ready datasets, so they’re built to support GenAI and ML use cases, not just BI.
• Design and evolve our medallion (Bronze/Silver/Gold) lakehouse architecture so it scales with the business.
• Set standards for data quality, security, and access control across the platform, and make sure teams actually follow them.
• Tune Spark jobs and cluster configs for performance and cost.
• Find the waste, right‑size the clusters, kill the idle spend.
• Build and maintain the core ETL frameworks and reusable components other teams build on top of.
• Lead migrations from legacy platforms onto Databricks, including data modeling and pipeline redesign.
• Embed infrastructure‑as‑code and CI/CD practices into how we build, deploy, and run pipelines and platform infrastructure, so deployments stay consistent and repeatable.
• Troubleshoot production issues, do root cause analysis, and put fixes in place that prevent repeat incidents.
• Mentor senior engineers and act as the technical escalation point across the Data teams.
• Set the standard for spec‑driven development and agentic AI tooling, then drive adoption for pipeline generation, testing, documentation, and incident response across the Data teams.
• Stay current on Databricks releases and bring in new capabilities when they solve a real problem, not just because they’re new.

What We’re Looking For

• 12+ years in data engineering, with at least 3+ years working hands‑on with Databricks in production.
• Deep expertise in Unity Catalog, Delta Lake, and Spark performance tuning at scale.
• Real experience designing and running medallion/lakehouse architectures.
• Strong SQL and Python.
• Experience with at least one major cloud platform (AWS, Azure, or GCP) and infrastructure‑as‑code tooling.
• Experience with CI/CD for data pipelines, including Databricks Asset Bundles.
• A track record of leading technical decisions and mentoring other engineers.
• Experience rolling out AI agents or AI‑assisted tooling across an engineering team. You’ve set standards and gotten a team to adopt them.
• Clear communicator who can work across engineering, analytics, and business stakeholders.

Nice to Have

• Experience with orchestration tools like Airflow or dbt.
• Experience with streaming/messaging systems like Kafka.
• Exposure to GenAI, RAG, or LLM‑based data applications.
• Experience leading a legacy data warehouse to Databricks migration.

What We Offer

• Comprehensive health benefits
• Generous paid time off, including paid holidays, volunteer days, quarterly self‑care days, and company‑designated no‑meeting days
• Tuition reimbursement and access to instructor‑led and on‑demand learning and development programs
• The Thrive Global Wellness Program, a confidential Employee Assistance Program (EAP), a wellness app and one‑on‑one wellness coaching
• Employee‑led communities and programs, including Employee Resource Groups (ERGs), GoTo Gives, and charitable matching

We work hard to create an environment where everyone feels welcome, respected, and able to contribute.

Building a culture of belonging isn't just something we talk about - it's part of how we work every day.

We’re committed to creating an inclusive space for everyone, because we know unique perspectives make us a stronger company and community.

Specific benefits and offerings may vary by country in line with local regulations and market practices.

At GoTo, you’ll find the flexibility, resources, and support you need to thrive—at work, at home, and everywhere in between. You’ll work towards a shared goal with an open‑minded, cohesive team that’s greater than the sum of its parts.

We’re committed to creating an inclusive space for everyone, because we know unique perspectives make us a stronger company and community.

We believe in being transparent about how AI is used in our hiring process and in keeping people at the center of every decision. Every interview, evaluation, and hiring decision is made by people. We also welcome the responsible use of AI during the application process — whether for resume editing, research, or interview preparation. What matters most is that your application reflects your authentic skills and perspective.

Learn more.