Senior Data Platform Engineer: AI Lakehouse

Hallmark Global Solutions Ltd; Cary (NC); On-site; USD 150,000 - 230,000; Full time

Job summary

Hallmark Global Solutions Ltd is seeking an experienced Data Platform Architect/Engineer to own the end-to-end lakehouse architecture across Bronze, Silver, and Gold layers. You will design parameterized ingestion frameworks and canonical mappings, and lead production-grade Spark deployments on Delta Lake with cost-conscious tuning.

You will also drive AI-augmented ingestion and data quality initiatives, build a semantic layer with a GPT-powered query interface, and mentor engineers while

Qualifications

• Expert-level proficiency in Python, Scala, and PySpark with production-grade solutions.
• Strong SQL and data modelling experience (dimensional and normalised).
• Databricks expertise including Delta Lake, Unity Catalog, Jobs & Workflows, and performance tuning.
• 3+ years designing and shipping LLM-based systems in production (RAG, embeddings, retrieval).
• Hands-on experience with LangChain, LlamaIndex, or LangGraph and provider stacks (Azure OpenAI, OpenAI, Databricks).
• Metadata-driven thinking: schema inference, lineage, catalogs, config-driven frameworks.
• 12–18 years of total experience in data engineering or related disciplines.

Responsibilities

• Own end-to-end lakehouse architecture across Bronze/Silver/Gold layers.
• Design AI-augmented ingestion and canonical mapping pipelines with validation gates.
• Build AI-driven data quality, anomaly detection, and testing suites for data products.
• Develop semantic layer and a GPT-powered conversational interface for data access.
• Lead architecture decisions, ensure governance, and mentor data engineers.

Skills

Python

Scala

PySpark

SQL

Databricks

Unity Catalog

LLM pipelines

LangChain

LangGraph

Azure OpenAI / OpenAI

Tools

Azure DevOps

Terraform

Databricks Asset Bundles

dbt / Great Expectations

Job description

Hallmark Global Solutions Ltd is seeking an experienced Data Platform Architect/Engineer to own the end-to-end lakehouse architecture across Bronze, Silver, and Gold layers. You will design parameterized ingestion frameworks and canonical mappings, and lead production-grade Spark deployments on Delta Lake with cost-conscious tuning.

You will also drive AI-augmented ingestion and data quality initiatives, build a semantic layer with a GPT-powered query interface, and mentor engineers while