Senior Data Engineer — LLM & Spark Platform (Delta Lake)

Altimetrik

Cary (NC)

On-site

USD 120,000 - 170,000

Full time

5 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Altimetrik is seeking an expert data engineering professional to design and deliver modular, production-ready pipelines using Python, Scala, PySpark, and Delta Lake on Databricks. You will optimize Spark workloads and contribute to large-scale batch and streaming data flows with robust data contracts.

You will build and evaluate LLM-based systems, including RAG pipelines and tool-calling workflows, leveraging LangChain or similar stacks and provider solutions like OpenAI or Azure OpenAI.

Qualifications

  • Expert in Python, Scala and PySpark with production-grade design skills.
  • Strong SQL and data modelling, with dimensional and normalised schemas.
  • Hands-on Databricks experience: Delta Lake, Unity Catalog, Jobs & Workflows.
  • 3+ years building LLM-based systems including RAG pipelines and tool-calling workflows.
  • Familiar with LangChain, LlamaIndex, or LangGraph and provider stacks (Azure OpenAI, OpenAI).

Responsibilities

  • Design and deliver modular, production-ready data pipelines.
  • Troubleshoot Spark workloads and optimize large-scale batch/streaming jobs.
  • Implement Delta Lake based architectures and data contracts.
  • Develop and evaluate LLM-based systems with quality controls and human-in-the-loop feedback.

Skills

Python
Scala
PySpark
SQL
Data modelling
Delta Lake
Databricks
Spark workloads
LangChain
LlamaIndex
LangGraph
Azure OpenAI
OpenAI
Model Serving
Metadata-driven thinking
Vector search

Education

Tools

Databricks
Delta Lake
Unity Catalog
Jobs & Workflows
Cluster management
Performance tuning
Databricks Model Serving

Job description

Altimetrik is seeking an expert data engineering professional to design and deliver modular, production-ready pipelines using Python, Scala, PySpark, and Delta Lake on Databricks. You will optimize Spark workloads and contribute to large-scale batch and streaming data flows with robust data contracts.

You will build and evaluate LLM-based systems, including RAG pipelines and tool-calling workflows, leveraging LangChain or similar stacks and provider solutions like OpenAI or Azure OpenAI.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Lead Data Engineer
Lead Data Engineer

Altimetrik • Cary (NC)

On-site
USD 120,000 - 170,000
Senior Databricks Engineer - Delta Lake & ML Ops
Senior Databricks Engineer - Delta Lake & ML Ops

Long Finch Technologies, LLC. • Galena (OH)

On-site
USD 140,000 - 170,000
Senior Data Infrastructure Engineer (Spark & Delta Lake)
Senior Data Infrastructure Engineer (Spark & Delta Lake)

Neon • Mountain View (CA)

On-site
USD 166,000 - 225,000
Senior Data Engineer - Databricks & Delta Lake (Remote)
Senior Data Engineer - Databricks & Delta Lake (Remote)

LorganTech • United States

Remote
USD 83,000 - 90,000
Senior Data Engineer - Lakehouse & ML Systems Lead
Senior Data Engineer - Lakehouse & ML Systems Lead

Raag Solutions LLC. • Cary (NC)

On-site
USD 120,000 - 180,000
Azure Data Platform Engineer: Delta Lake & Spark Expert
Azure Data Platform Engineer: Delta Lake & Spark Expert

Techwish • New York (NY)

On-site
USD 140,000 - 190,000
Senior Data Engineer
Senior Data Engineer

The Modulae • United States

Remote
USD 130,000 - 180,000
Remote work
Sr. Databricks Developer
Sr. Databricks Developer

Valiant Technologies LLC • Columbus (OH)

On-site
USD 120,000 - 190,000
Senior Data Engineer
Senior Data Engineer

LorganTech • United States

Remote
USD 83,000 - 90,000
Sr. Databricks Developer
Sr. Databricks Developer

Quantum Technologies LLC • Columbus (OH)

On-site
USD 120,000 - 180,000