Senior Data Platform Architect – Lakehouse & AI

Smart IT Frame LLC; Cary (NC); Hybride; USD 150 000 - 210 000; Plein temps

Résumé du poste

Smart IT Frame LLC seeks a seasoned Data Platform Architect to own end-to-end lakehouse design across Bronze/Silver/Gold layers. You will craft scalable ingestion pipelines, define canonical metadata, and implement robust Spark-based transformations.

You will optimize Spark performance, manage cost at scale, and automate CI/CD for Databricks and Azure DevOps, ensuring reliable data delivery and observability.

Qualifications

• Expert-level proficiency in Python, Scala, and PySpark with production experience.
• Strong SQL and data modelling skills, including dimensional design.
• Databricks expertise: Delta Lake, Unity Catalog, Jobs & Workflows, performance tuning.
• 3+ years designing and shipping LLM-based systems in production: RAG, tool-calling workflows.
• CI/CD and IaC experience: Azure DevOps, Terraform, Databricks Asset Bundles.

Responsabilités

• Own end-to-end lakehouse architecture: Bronze/Silver/Gold, ADLS Gen2 layout, Delta table design.
• Design metadata-driven ingestion frameworks to onboard new data sources without bespoke code.
• Write production PySpark/Scala jobs; establish coding/testing standards and review PRs.
• Tune Spark clusters, apply partition pruning and caching; manage cost as data grows.
• Automate CI/CD for Databricks and ADF pipelines; maintain observability with Azure Monitor.

Connaissances

Python

Scala

PySpark

SQL

Data modeling

Delta Lake

Spark tuning

Outils

Databricks

Unity Catalog

Azure DevOps

Terraform

Azure Monitor

Description du poste

Smart IT Frame LLC seeks a seasoned Data Platform Architect to own end-to-end lakehouse design across Bronze/Silver/Gold layers. You will craft scalable ingestion pipelines, define canonical metadata, and implement robust Spark-based transformations.

You will optimize Spark performance, manage cost at scale, and automate CI/CD for Databricks and Azure DevOps, ensuring reliable data delivery and observability.