Lead Data Engineer: Lakehouse Platform & AI Pipelines

Megthink; Cary (NC); On-site; USD 150,000 - 210,000; Full time

Job summary

Megthink seeks a senior data platform engineer to architect and deliver lakehouse solutions across Bronze to Gold layers. You will own production-grade pipelines using PySpark/Scala, Delta Lake, and Unity Catalog, while shaping governance and security at scale.

In this role you will mentor engineers, implement CI/CD for Databricks and ADF, and advance AI-powered data ingestion and semantic access. Experience with LangChain and OpenAI stacks is preferred.

Qualifications

• Expert-level Python, Scala and PySpark for production-ready, modular pipelines.
• Strong SQL and data modeling, data contracts and dimensional modeling.
• Databricks expertise: Delta Lake, Unity Catalog, Jobs & Workflows, performance tuning.
• 3+ years shipping LLM-based systems in production (RAG, tool-calling, embeddings).
• Experience with LangChain or similar tools and provider stacks (Azure OpenAI/OpenAI).
• 12-18 years of total experience in data engineering / data platform delivery.
• Proven enterprise-scale delivery of medallion/lakehouse architectures.
• Azure security, RBAC, Key Vault, private endpoints, PII handling.

Responsibilities

• Design and deliver lakehouse architectures (Bronze/Silver/Gold) and durable data contracts.
• Lead and mentor engineers; drive documentation and governance practices.
• Build and maintain CI/CD pipelines for Databricks and ADF; ensure observability.
• Implement AI-augmented ingestion, quality checks, and schema-driven pipelines.
• Develop semantic layers and knowledge graphs with secure data access.

Skills

Python/Scala

PySpark

SQL/Data Modeling

Databricks

LLM Systems

CI/CD

IaC

LangChain

Data Engineering

Tools

Delta Lake

Unity Catalog

Azure DevOps

Terraform

Databricks

LangChain

LlamaIndex

OpenAI

Job description

Megthink seeks a senior data platform engineer to architect and deliver lakehouse solutions across Bronze to Gold layers. You will own production-grade pipelines using PySpark/Scala, Delta Lake, and Unity Catalog, while shaping governance and security at scale.

In this role you will mentor engineers, implement CI/CD for Databricks and ADF, and advance AI-powered data ingestion and semantic access. Experience with LangChain and OpenAI stacks is preferred.