Manager - Data Engineering

Latent View Analytics Limited; Hinoba-an; On-site; PHP 1,200,000 - 2,400,000; Full time

Job summary

Latent View Analytics Limited in the Philippines seeks a Senior Data Engineer focused on building AI-first infrastructure atop the Databricks stack. You will architect Lakehouse with Bronze/Silver/Gold layers and lead data pipelines for ML and LLM workloads, while enabling internal teams with self-service tooling.

You will optimize Unity Catalog governance, deploy Delta Live Tables, manage serverless SQL compute, and ensure scalable, low-latency data delivery for customer-facing AI features.

Qualifications

• Senior data engineering resource with a focus on infrastructure for ML/AI workloads.
• Experience designing and implementing lakehouse architectures with bronze/silver/gold layers.
• Expertise in Databricks stack including Unity Catalog and DLT.

Responsibilities

• AI-Centric Infrastructure Design: architect lakehouse and feature-first pipelines.
• Databricks Stack Mastery: governance with Unity Catalog and DLT deployment.
• Customer Enablement & Scalability: build self-service tooling for internal DS/ML teams.
• AI Operations (DataOps & MLOps): CI/CD for infra as code and data pipelines.
• Agentic framework support: enable LangChain or LlamaIndex workflows.

Skills

Databricks stack

AI-first infrastructure

Lakehouse architecture

Unity Catalog

Delta Live Tables

Vector databases

LangChain

LlamaIndex

Spark

Infrastructure as code

Tools

Terraform

Pulumi

Great Expectations

Monte Carlo

Databricks Unity Catalog

Job description

Job Description:

We need a senior data engineering resource who is super deep into building the infrastructure layer, has expertise in the Databricks stack and thinks AI first in terms building out the infrastructure - We are looking for this person as a Senior leader for the customer enablement stack who can support and help us get to the next level while building a truly AI centric stack for us.

Responsibilities:

1. AI-Centric Infrastructure Design

Architecting the Lakehouse: Lead the design and implementation of a robust Medallion Architecture (Bronze/Silver/Gold) specifically optimized for downstream machine learning and LLM consumption.

Vector Database Integration: Architect the seamless integration of Databricks Vector Search and managed vector databases to support RAG-based applications.

Model-Ready Pipelines: Build "feature-first" data pipelines where data is versioned, lineage-tracked, and ready for training without manual preprocessing.

2. Databricks Stack Mastery

Unity Catalog Governance: Implement enterprise-wide data governance, security, and discovery using Unity Catalog to ensure AI models access data ethically and securely.

Delta Live Tables (DLT): Deploy and manage complex, streaming data pipelines using DLT to reduce operational overhead and increase data reliability.

Compute Optimization: Manage and optimize Serverless SQL warehouses and automated cluster scaling to balance high performance with cost-efficiency.

3. Customer Enablement & Scalability

Internal Productization: Treat the data stack as a product, building self-service tooling that allows internal "customers" (DS/ML teams) to spin up environments and access clean data instantly.

Performance Engineering: Debug and resolve deep-seated architectural bottlenecks in Spark jobs to ensure sub-second latency for customer-facing AI features.

Technical Evangelism: Act as the bridge between core engineering and customer-facing teams, translating complex infrastructure capabilities into business value.

4. AI Operations (DataOps & MLOps)

CI/CD for Data: Establish rigorous CI/CD practices for infrastructure-as-code (Terraform/Pulumi) and data pipeline deployments.

Monitoring & Observability: Implement advanced monitoring for data quality (Great Expectations/Monte Carlo) and model drift, ensuring the AI stack is "self-healing."

Agentic Framework Support: Design the backend infra to support LangChain or LlamaIndex workflows, ensuring the data retrieval layer is fast enough for agentic reasoning.