Lead Data Engineer

Raag Solutions LLC

Cary (NC)

On-site

USD 120,000 - 160,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Raag Solutions LLC is seeking a senior data engineer with extensive experience in building enterprise-grade lakehouse architectures. The role focuses on Python/Scala/PySpark, large-scale batch and streaming data pipelines, and production-grade data platforms using Delta Lake and Databricks.

You will design data contracts, governance, and metadata-driven frameworks, while collaborating on AI/ML workloads, LLM-based systems, and MLOps practices across Azure OpenAI or OpenAI provider stacks.

Qualifications

  • 12-18 years of total experience in data engineering, data platform delivery, or related disciplines.
  • Proven delivery of a medallion / lakehouse architecture at enterprise scale — not just familiarity with the concept.
  • Strong proficiency in Python, Scala and PySpark with production-grade pipelines.
  • Extensive experience with Databricks, Delta Lake and Unity Catalog.
  • Deep knowledge of Azure data stack: ADLS Gen2, Data Factory, Event Hubs.
  • Hands-on with LangChain, LlamaIndex, or LangGraph and at least one provider (Azure OpenAI/OpenAI/Databricks Model Serving).
  • Strong SQL, data modelling (dimensional and normalised) and data contracts.
  • Experience designing and shipping AI/ML systems in production: RAG pipelines, prompt engineering, vector stores.
  • Experience with MLOps: model/versioning, cost governance, observability.
  • Security and governance: RBAC, PII handling, Entra ID, private endpoints.

Skills

Python
Scala
PySpark
SQL
Data modeling
Dimensional modeling
Normalised schemas
Data contracts
AI/ML systems
LLM-based systems
RAG pipelines
LangChain
LlamaIndex
LangGraph
Knowledge graphs
Ontologies
Semantic queries
Time-series analytics
Time-series anomaly detection
Data quality
Golden datasets
Human-in-the-loop
Model serving
Azure security
RBAC
PII handling
DevOps
Terraform
Databricks
Delta Lake
Unity Catalog
Azure Data Factory
Azure Data Lake
Azure Event Hubs
Databricks Model Serving
dbt
Great Expectations
Workday/Prism exposure
Data governance

Tools

Databricks
Delta Lake
Unity Catalog
Jobs & Workflows
Cluster & pool management
performance tuning
Azure Data Factory
ADLS Gen2
Azure Event Hubs
LangChain
LlamaIndex
LangGraph
Azure OpenAI
OpenAI
Databricks Model Serving
dbt
Great Expectations
Workday
Prism
Terraform

Job description

MUST-HAVE SKILLS & EXPERIENCE
  • Programming & Data Engineering Expert - level proficiency in Python, Scala, and PySpark, with a strong track record of designing and delivering production-ready, modular, and well-tested solutions; developing and troubleshooting Spark workloads; and optimizing large-scale batch and streaming data pipelines using Delta Lake and Spark technologies.
  • Strong SQL and data modelling — dimensional and normalised; schema design and data contract definition.
  • Databricks expertise — Delta Lake, Unity Catalog, Jobs & Workflows, cluster and pool management, performance tuning, Model Serving.
  • Azure data stack — ADLS Gen2 (zone design, ACLs, lifecycle), Azure Data Factory (parameterized / metadata-driven frameworks, error handling), Azure Event Hubs.
  • AI & Machine Learning 3+ years designing and shipping LLM-based systems in production: RAG pipelines, agentic / tool-calling workflows, structured output, chunking and embedding strategy, vector and hybrid retrieval, and prompt engineering.
  • Evaluation discipline — golden datasets, regression suites, accuracy and hallucination tracking, human-in-the-loop feedback loops; you measure AI quality, not assert it.
  • Hands-on experience with LangChain, LlamaIndex, or LangGraph, plus at least one provider stack (Azure OpenAI, OpenAI, or Databricks Model Serving).
  • Metadata-driven thinking — schema inference, data profiling, lineage, catalogs, and configuration-driven frameworks that onboard the next source without new code.
  • Architecture & Governance 12-18 years of total experience in data engineering, data platform delivery, or related disciplines.
  • Proven delivery of a medallion / lakehouse architecture at enterprise scale — not just familiarity with the concept.
  • Azure security and governance — Entra ID, managed identities, RBAC, POSIX ACLs on ADLS Gen2, Key Vault, private endpoints, and PII handling.
  • CI/CD and infrastructure as code — Azure DevOps, Terraform, Databricks Asset Bundles, and automated testing of data pipelines.
  • Clear technical writing and the ability to present and defend a design to both engineers and non-technical stakeholders.
  • STRONGLY PREFERRED Knowledge graphs and ontologies: RDF/SPARQL, property graphs (Neo4j), or graph modelling over a lakehouse.
  • Text-to-SQL or semantic-layer-backed natural-language query systems at enterprise scale, including access control and ambiguity handling.
  • ML-based anomaly detection on time-series or transactional financial data.
  • Financial services or insurance domain knowledge: finance close, general ledger, subledger, reconciliation, or actuarial data.
  • LLMOps and MLOps: model versioning, prompt versioning, cost governance, and observability tooling.
  • Databricks Data Engineer Professional, Azure DP-203 / DP-700 / AZ-305 certification.
  • dbt, Great Expectations, or similar data-quality and transformation tooling.
  • Workday, Prism, or Accounting Center exposure.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Lead Data Engineer
Lead Data Engineer

Raag Solutions • Cary (NC)

On-site
USD 120,000 - 180,000
Lead Data Engineer
Lead Data Engineer

Altimetrik • Cary (NC)

On-site
USD 120,000 - 170,000
Databricks Data Architect
Databricks Data Architect

NewVision • San Francisco (CA)

Hybrid
USD 140,000 - 190,000
Hybrid work model
Lead Databricks Data Engineer
Lead Databricks Data Engineer

Fa Ewjt Saasfaprod1 • New York (NY)

On-site
USD 140,000 - 170,000
Senior Databricks Engineer
Senior Databricks Engineer

Remote • San Francisco (CA)

On-site
USD 150,000 - 190,000
Databricks Developer
Databricks Developer

Vservice It Solutions • United States

Remote
USD 140,000 - 190,000
Databricks Data Architect
Databricks Data Architect

NewVision Software • United States

Remote
USD 120,000 - 180,000
Azure Databricks Data Architect
Azure Databricks Data Architect

Ascendum Solutions • Cincinnati (OH)

On-site
USD 140,000 - 170,000
Lead Databricks Data Engineer
Lead Databricks Data Engineer

EXL Talent Acquisition Team • New York (NY)

On-site
USD 180,000 - 240,000
Azure Data Engineer
Azure Data Engineer

TechDigital Group • Pleasanton (CA)

On-site
USD 120,000 - 170,000