Lead Data Engineer

Altimetrik

Cary (NC)

On-site

USD 120,000 - 170,000

Full time

5 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Altimetrik is seeking an expert data engineering professional to design and deliver modular, production-ready pipelines using Python, Scala, PySpark, and Delta Lake on Databricks. You will optimize Spark workloads and contribute to large-scale batch and streaming data flows with robust data contracts.

You will build and evaluate LLM-based systems, including RAG pipelines and tool-calling workflows, leveraging LangChain or similar stacks and provider solutions like OpenAI or Azure OpenAI.

Qualifications

  • Expert in Python, Scala and PySpark with production-grade design skills.
  • Strong SQL and data modelling, with dimensional and normalised schemas.
  • Hands-on Databricks experience: Delta Lake, Unity Catalog, Jobs & Workflows.
  • 3+ years building LLM-based systems including RAG pipelines and tool-calling workflows.
  • Familiar with LangChain, LlamaIndex, or LangGraph and provider stacks (Azure OpenAI, OpenAI).

Responsibilities

  • Design and deliver modular, production-ready data pipelines.
  • Troubleshoot Spark workloads and optimize large-scale batch/streaming jobs.
  • Implement Delta Lake based architectures and data contracts.
  • Develop and evaluate LLM-based systems with quality controls and human-in-the-loop feedback.

Skills

Python
Scala
PySpark
SQL
Data modelling
Delta Lake
Databricks
Spark workloads
LangChain
LlamaIndex
LangGraph
Azure OpenAI
OpenAI
Model Serving
Metadata-driven thinking
Vector search

Education

Tools

Databricks
Delta Lake
Unity Catalog
Jobs & Workflows
Cluster management
Performance tuning
Databricks Model Serving

Job description

Expert - level proficiency in Python, Scala, and PySpark, with a strong track record of designing and delivering production-ready, modular, and well-tested solutions; developing and troubleshooting Spark workloads; and optimizing large-scale batch and streaming data pipelines using Delta Lake and Spark technologies.

Strong SQL and data modelling — dimensional and normalised; schema design and data contract definition.

Databricks expertise — Delta Lake, Unity Catalog, Jobs & Workflows, cluster and pool management, performance tuning, Model Serving.

3+ years designing and shipping LLM-based systems in production: RAG pipelines, agentic / tool-calling workflows, structured output, chunking and embedding strategy, vector and hybrid retrieval, and prompt engineering.

Evaluation discipline — golden datasets, regression suites, accuracy and hallucination tracking, human-in-the-loop feedback loops; you measure AI quality, not assert it.

Hands-on experience with LangChain, LlamaIndex, or LangGraph, plus at least one provider stack (Azure OpenAI, OpenAI, or Databricks Model Serving).

Metadata-driven thinking — schema inference, data profiling, lineage, catalogs, and configuration-driven frameworks that onboard the next source without new code.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Lead Data Engineer
Lead Data Engineer

Raag Solutions LLC • Cary (NC)

On-site
USD 120,000 - 160,000
Lead Data Engineer
Lead Data Engineer

Raag Solutions • Cary (NC)

On-site
USD 120,000 - 180,000
Senior Data Engineer — LLM & Spark Platform (Delta Lake)
Senior Data Engineer — LLM & Spark Platform (Delta Lake)

Altimetrik • Cary (NC)

On-site
USD 120,000 - 170,000
Lead Data Engineer – Databricks
Lead Data Engineer – Databricks

TechDigital Group • Bolingbrook (IL)

On-site
USD 120,000 - 150,000
Senior Databricks Engineer
Senior Databricks Engineer

Remote • San Francisco (CA)

On-site
USD 150,000 - 190,000
Data Engineer
Data Engineer

Resolve Tech Solutions • Irving (TX)

On-site
USD 110,000 - 150,000
Data Engineer
Data Engineer

GSquared Group • Atlanta (GA)

On-site
USD 120,000 - 160,000
Lead Data Engineer - Databricks & Spark Expert
Lead Data Engineer - Databricks & Spark Expert

Link Group • United States

On-site
USD 140,000 - 200,000
Senior Data Engineer
Senior Data Engineer

Veriipro • Atlanta (GA)

On-site
USD 140,000 - 180,000
Databricks Developer
Databricks Developer

Vservice It Solutions • United States

Remote
USD 140,000 - 190,000