Lead Data Engineer

Smart IT Frame LLC; Cary (NC); Hybrid; USD 150,000 - 210,000; Full time

Job summary

Smart IT Frame LLC seeks a seasoned Data Platform Architect to own end-to-end lakehouse design across Bronze/Silver/Gold layers. You will craft scalable ingestion pipelines, define canonical metadata, and implement robust Spark-based transformations.

You will optimize Spark performance, manage cost at scale, and automate CI/CD for Databricks and Azure DevOps, ensuring reliable data delivery and observability.

Qualifications

• Expert-level proficiency in Python, Scala, and PySpark with production experience.
• Strong SQL and data modelling skills, including dimensional design.
• Databricks expertise: Delta Lake, Unity Catalog, Jobs & Workflows, performance tuning.
• 3+ years designing and shipping LLM-based systems in production: RAG, tool-calling workflows.
• CI/CD and IaC experience: Azure DevOps, Terraform, Databricks Asset Bundles.

Responsibilities

• Own end-to-end lakehouse architecture: Bronze/Silver/Gold, ADLS Gen2 layout, Delta table design.
• Design metadata-driven ingestion frameworks to onboard new data sources without bespoke code.
• Write production PySpark/Scala jobs; establish coding/testing standards and review PRs.
• Tune Spark clusters, apply partition pruning and caching; manage cost as data grows.
• Automate CI/CD for Databricks and ADF pipelines; maintain observability with Azure Monitor.

Skills

Python

Scala

PySpark

SQL

Data modeling

Delta Lake

Spark tuning

Tools

Databricks

Unity Catalog

Azure DevOps

Terraform

Azure Monitor

Job description

Location : Cary, NC (On-site / Hybrid)

Duration : Full Time

Experience:

12-18 Years of Experience

Data Platform Architecture & Engineering

Own the end-to-end lakehouse architecture: Bronze / Silver / Gold layer contracts, zone layout on ADLS Gen2, Delta Lake table design, partitioning, schema evolution, and retention strategy.

Design and build metadata-driven, parameterized ingestion frameworks that onboard new data sources without bespoke pipeline code for every feed.

Write canonical PySpark and Scala Spark transformation jobs that serve as the team reference; set coding and testing standards, review pull requests, and debug production incidents.

Design for scale and cost: tune Spark clusters and pools, apply partition pruning and caching strategies, and set cost guardrails as data volumes grow.

Build and automate CI/CD for Databricks and ADF pipelines in Azure DevOps using Databricks Asset Bundles and Terraform; maintain platform observability with Azure Monitor and Log Analytics.

Design repeatable patterns for batch files, database extracts, CDC feeds, and streaming ingestion using Azure Event Hubs / Kafka and Spark Structured Streaming.

AI-Augmented Ingestion & Canonical Mapping

Design and build the AI-augmented metadata ingestion framework that auto-generates bridge documents, DML statements, canonical table definitions, and control metadata from source schemas.

Build AI-assisted source-to-canonical attribute mapping: schema reasoning, data profiling, confidence scoring, and a human review / approval gate before any mapping reaches production.

Generate file-level and record-level validation rules from historical data and metadata analysis, feeding a configurable, rules-engine-backed data quality framework.

AI-Driven Data Quality, Anomaly Detection & Testing

Semantic Layer & Conversational Data Access

MUST-HAVE SKILLS & EXPERIENCE

Expert - level proficiency in Python, Scala, and PySpark, with a strong track record of designing and delivering production-ready, modular, and well-tested solutions; developing and troubleshooting Spark workloads; and optimizing large-scale batch and streaming data pipelines using Delta Lake and Spark technologies.

Strong SQL and data modelling - dimensional and normalised; schema design and data contract definition.

Databricks expertise - Delta Lake, Unity Catalog, Jobs & Workflows, cluster and pool management, performance tuning, Model Serving.

3+ years designing and shipping LLM-based systems in production: RAG pipelines, agentic / tool-calling workflows, structured output, chunking and embedding strategy, vector and hybrid retrieval, and prompt engineering.

Evaluation discipline - golden datasets, regression suites, accuracy and hallucination tracking, human-in-the-loop feedback loops; you measure AI quality, not assert it.

Hands-on experience with LangChain, LlamaIndex, or LangGraph, plus at least one provider stack (Azure OpenAI, OpenAI, or Databricks Model Serving).

Metadata-driven thinking - schema inference, data profiling, lineage, catalogs, and configuration-driven frameworks that onboard the next source without new code.

12-18 years of total experience in data engineering, data platform delivery, or related disciplines.

Proven delivery of a medallion / lakehouse architecture at enterprise scale - not just familiarity with the concept.

Azure security and governance - Entra ID, managed identities, RBAC, POSIX ACLs on ADLS Gen2, Key Vault, private endpoints, and PII handling.

CI/CD and infrastructure as code - Azure DevOps, Terraform, Databricks Asset Bundles, and automated testing of data pipelines.

Clear technical writing and the ability to present and defend a design to both engineers and non-technical stakeholders.

STRONGLY PREFERRED

Knowledge graphs and ontologies: RDF/SPARQL, property graphs (Neo4j), or graph modelling over a lakehouse.

Text-to-SQL or semantic-layer-backed natural-language query systems at enterprise scale, including access control and ambiguity handling.

ML-based anomaly detection on time-series or transactional financial data.

Financial services or insurance domain knowledge: finance close, general ledger, subledger, reconciliation, or actuarial data.

LLMOps and MLOps: model versioning, prompt versioning, cost governance, and observability tooling.

Databricks Data Engineer Professional, Azure DP-203 / DP-700, or AZ-305 certification.

dbt, Great Expectations, or similar data-quality and transformation tooling.

Workday, Prism, or Accounting Center exposure.