Principal AI-Data Engineer

Globallogic; Hyderabad; Hybrid; INR 4,000,000 - 6,000,000; Full time

Benefits offered by this job

Flexible work schedules

Work from home options

Paid time off

Job summary

Globallogic is seeking a Principal Engineer to lead architecture and execution of an enterprise AI Data Control Plane. You will design a multi-cloud data virtualization layer using Databricks Unity Catalog and expose governed data access to AI agents via MCP.

The role requires deep expertise in Databricks Lakehouse Federation, MCP, and secure cross-cloud data access, with a focus on scalable, low-leakage credential management.

Qualifications

• 8+ years of distributed data systems architecture.
• 3+ years specifically in Databricks, Unity Catalog, and multi-cloud data environments.
• Deep expertise in Databricks Lakehouse Federation and query optimization across BigQuery, Snowflake, and PostgreSQL.
• Advanced Python proficiency with databricks-sdk, databricks-sql-connector, pydantic, asyncio, FastAPI.
• Hands-on experience with LLM tool calling, MCP, or building custom agentic data wrappers.

Responsibilities

• Design and deploy multi-cloud data virtualization using Databricks Lakehouse Federation across GCP, AWS, and Azure.
• Build MCP servers to expose Unity Catalog primitives to external LLM agents.
• Implement row-level security and column masking in Unity Catalog to secure federated data.
• Tune Serverless SQL Warehouses and Materialized Views for high-concurrency workloads.

Skills

Databricks Lakehouse Federation

SQL pushdown

Python (databricks-sdk, databricks-sql

LLM tool calling

OAuth 2.0 / IAM

Terraform

CI/CD pipelines

Multi-cloud networking

Education

Databricks Certified Data Engineer Professional

Databricks Certified Machine Learning Professional

Tools

Databricks Genie

LangChain

LlamaIndex

PySpark

Databricks Asset Bundles

Job description

Description:

Principal Engineer – AI Data Control Plane & Lakehouse Federation

Role Overview

We are looking for a Principal Engineer to lead the architectural design and execution of our enterprise AI Data Control Plane. In this role, you will architect a multi-cloud data virtualization layer using Databricks Unity Catalog (Lakehouse Federation) across disparate systems (BigQuery, Snowflake, PostgreSQL) and expose governed, semantic data access to AI agents via the Model Context Protocol (MCP). You will act as the technical authority bridging data engineering, cloud infrastructure, and AI systems—enabling next-generation AI agents and applications to query enterprise data safely, performantly, and without credential leakage.

Requirements

• Experience: 8+ years of distributed data systems architecture, with 3+ years specifically in Databricks, Unity Catalog, and multi-cloud data environments.
• Core Technical Mastery: Deep expertise in Databricks Lakehouse Federation, SQL pushdown mechanics, and query optimization across BigQuery, Snowflake, and PostgreSQL.
• Programming & SDKs: Advanced Python proficiency (databricks-sdk, databricks-sql-connector, pydantic, asyncio, FastAPI).
• AI & Agentic Systems: Hands-on experience with LLM tool calling, Model Context Protocol (MCP), Databricks Genie, or building custom agentic data wrappers.
• Cloud & Networking: Strong understanding of multi-cloud network topologies (AWS PrivateLink, GCP PSC, Azure Private Endpoints) and credential management (OAuth 2.0, Key Vault, IAM).
• Infrastructure as Code: Proficiency with Terraform and automated CI/CD pipelines for data platform deployment.
• ++++
• Prior experience building an internal developer platform or API control plane for AI agents.
• Active contributions to open-source data or AI tools (e.g., Anthropic MCP SDKs, LangChain, LlamaIndex, PySpark).
• Databricks Certified Data Engineer Professional or Databricks Certified Machine Learning Professional.

Job Responsibilities

• Multi-Cloud Data Virtualization Architecture: Design and deploy regional Databricks Lakehouse Federation topologies across GCP, AWS, and Azure (connecting BigQuery, Snowflake, and PostgreSQL). Optimize query pushdown execution, parallel JDBC read slicing, and network routing to minimize cross-cloud egress costs and query latency.
• MCP & AI Middleware Platform: Architect and build high-performance, Python-based MCP Servers (API Faade, Router, and Metadata Registry) that expose Unity Catalog primitives as AI tools to external LLM agents. Implement semantic catalog indexing using Databricks Vector Search and DatabricksIQ (ai_query) to auto-enrich table/column metadata dictionaries.
• Data Governance & Security: Implement row-level security (RLS) filters and column masking rules directly inside Unity Catalog to secure federated data. Enforce zero-credential leakage using machine-to-machine OAuth, Databricks Secret Scopes, and system-level audit logs (system.access.audit).
• Platform Performance & Reliability: Configure and tune Serverless SQL Warehouses and Materialized Views for high-concurrency, low-latency agentic workloads. Establish CI/CD, GitOps, and IaC frameworks (Terraform, Databricks Asset Bundles) for automated change management across Dev, QA, and Prod environments.

What We Offer

Exciting Projects: We focus on industries like High-Tech, communication, media, healthcare, retail and telecom. Our customer list is full of fantastic global brands and leaders who love what we build for them.

Collaborative Environment: You Can expand your skills by collaborating with a diverse team of highly talented people in an open, laidback environment — or even abroad in one of our global centers or client facilities!

Work-Life Balance: GlobalLogic prioritizes work-life balance, which is why we offer flexible work schedules, opportunities to work from home, and paid time off and holidays.

Professional Development: Our dedicated Learning & Development team regularly organizes Communication skills training(GL Vantage, Toast Master),Stress Management program, professional certifications, and technical and soft skill trainings.

Excellent Benefits: We provide our employees with competitive salaries, family medical insurance, Group Term Life Insurance, Group Personal Accident Insurance , NPS(National Pension Scheme ), Periodic health awareness program, extended maternity leave, annual performance bonuses, and referral bonuses.

Fun Perks: We want you to love where you work, which is why we host sports events, cultural activities, offer food on subsidies rates, Corporate parties. Our vibrant offices also include dedicated GL Zones, rooftop decks and GL Club where you can drink coffee or tea with your colleagues over a game of table and offer discounts for popular stores and restaurants!