Senior Lakehouse Data Engineer — AI-Driven Data Platform

Smart IT Frame LLC; Cary (NC); Híbrido; USD 180.000 - 240.000; Jornada completa

Descripción de la vacante

Smart IT Frame LLC is seeking a senior data platform architect to own end-to-end lakehouse architectures and AI-augmented ingestion pipelines. You will design scalable, cost-aware data platforms, implement metadata-driven frameworks, and lead the creation of AI-assisted data quality and semantic access layers.

You will mentor engineers, define standards, and ensure production-grade, testable pipelines across Bronze/Silver/Gold layers in a hybrid on-site environment in Cary, NC.

Formación

• Expert-level proficiency in Python, Scala, and PySpark for production-grade data pipelines.
• Strong SQL data modeling and schema design.
• Hands-on Databricks Delta Lake and Unity Catalog expertise.
• 12–18 years of total data engineering/ platform delivery experience.
• Experience with LangChain, LlamaIndex, or LangGraph and cloud AI stacks.
• CI/CD and IaC with Azure DevOps and Terraform.

Responsabilidades

• Own end-to-end lakehouse architecture, Bronze/Silver/Gold layer contracts, partitioning, and retention strategies.
• Design metadata-driven ingestion frameworks that onboard new data sources without bespoke code.
• Write canonical PySpark/Scala Spark transformations and set coding/testing standards.
• Design for scale and cost with Spark tuning, partition pruning, and caching.
• Automate CI/CD for Databricks and ADF pipelines using Databricks Asset Bundles and Terraform.
• Build AI-augmented ingestion framework that auto-generates bridge documents and canonical definitions.
• Develop AI-assisted source-to-canonical attribute mapping with review gates.
• Create AI-driven data quality rules, anomaly detection, and automated test data generation.
• Design a semantic layer and GPT-powered data access interface with secure, source-traceable results.
• Define architecture governance, AI governance, and maintain observability across platforms.
• Mentor engineers, lead design reviews, and produce reusable components.

Conocimientos

Python

Scala

PySpark

SQL Modeling

Delta Lake

Unity Catalog

Databricks

LLM Pipelines

LangChain

LlamaIndex

LangGraph

Azure OpenAI

Model Serving

Metadata

Azure DevOps

Terraform

Asset Bundles

Data Governance

Herramientas

Azure DevOps

Terraform

Databricks

Asset Bundles

Unity Catalog

Descripción del empleo

Smart IT Frame LLC is seeking a senior data platform architect to own end-to-end lakehouse architectures and AI-augmented ingestion pipelines. You will design scalable, cost-aware data platforms, implement metadata-driven frameworks, and lead the creation of AI-assisted data quality and semantic access layers.

You will mentor engineers, define standards, and ensure production-grade, testable pipelines across Bronze/Silver/Gold layers in a hybrid on-site environment in Cary, NC.