Assistant Manager - Data Engineering

LatentView Analytics Ltd.; Tamil Nadu; Híbrido; INR 420.000 - 720.000; Jornada completa

Descripción de la vacante

LatentView Analytics Ltd. in India (Tamil Nadu) seeks an Assistant Manager - Data Engineering with 7–10 years of experience to lead AI-centric infrastructure initiatives. You will architect large-scale lakehouse solutions, integrate Databricks vector databases, and design feature-first pipelines for ML readiness.

You will champion governance with Unity Catalog, deploy Delta Live Tables, and drive DataOps/MLOps practices, ensuring scalable, secure data access for DS/ML teams.

Formación

• Senior data engineering resource with deep infra focus and AI-first mindset.
• Expertise in Databricks stack and building scalable infrastructure for ML/AI.
• Experience designing lakehouse architectures (Bronze/Silver/Gold) for AI/ML workloads.

Responsabilidades

• AI-Centric Infrastructure Design: Architect Lakehouse and MCP connectors.
• Databricks Stack Mastery: Unity Catalog governance and Delta Live Tables pipelines.
• Customer Enablement & Scalability: build self-service data tooling for DS/ML teams.
• AI Operations (DataOps & MLOps): CI/CD for infra, data pipelines, and observability.

Conocimientos

Databricks stack

Lakehouse architecture

AI-centric infra

MCP connectors

Vector databases

Delta Live Tables

CI/CD for data

Data quality monitoring

LangChain/LlamaIndex

Herramientas

Unity Catalog

Delta Live Tables

Terraform

Pulumi

Great Expectations

Monte Carlo

LangChain

LlamaIndex

Descripción del empleo

Designation : Assistant Manager - Data Engineering

Level : L3

Experience : 7 to 10 Years

Job Role:

We need a senior data engineering resource who is super deep into building the infrastructure layer, has expertise in the Databricks stack and thinks AI first in terms building out the infrastructure and MCPs required- We are looking for this person as a Senior leader for the Marketing Analytics stack who can support and help us get to the next level while building a truly AI centric stack for us.

Responsibilities

• 1. AI-Centric Infrastructure Design
  • Architecting the Lakehouse: Lead the design and implementation of a robust Medallion Architecture (Bronze/Silver/Gold) specifically optimized for downstream machine learning and LLM consumption.
  • Vector Database Integration: Architect the seamless integration of Databricks Vector Search and managed vector databases to support RAG-based applications.
  • Model-Ready Pipelines: Build "feature-first" data pipelines where data is versioned, lineage-tracked, and ready for training without manual preprocessing.
  • MCP: Build MCP connectors specific to the Databricks Marketing Analytics team
• 2. Databricks Stack Mastery
  • Unity Catalog Governance: Implement enterprise-wide data governance, security, and discovery using Unity Catalog to ensure AI models access data ethically and securely.
  • Delta Live Tables (DLT): Deploy and manage complex, streaming data pipelines using DLT to reduce operational overhead and increase data reliability.
• 3. Customer Enablement & Scalability
  • Internal Productization: Treat the data stack as a product, building self-service tooling that allows internal "customers" (DS/ML teams) to spin up environments and access clean data instantly.
  • Performance Engineering: Debug and resolve deep-stated architectural bottlenecks in Spark jobs to ensure sub-second latency for customer-facing AI features.
  • Technical Evangelism: Act as the bridge between core engineering and customer-facing teams, translating complex infrastructure capabilities into business value.
• 4. AI Operations (DataOps & MLOps)
  • CI/CD for Data: Establish rigorous CI/CD practices for infrastructure-as-code (Terraform/Pulumi) and data pipeline deployments.
  • Monitoring & Observability: Implement advanced monitoring for data quality (Great Expectations/Monte Carlo) and model drift, ensuring the AI stack is "self-healing."
  • Agentic Framework Support: Design the backend infra to support LangChain or LlamaIndex workflows, ensuring the data retrieval layer is fast enough for agentic reasoning.