Senior Data Engineer: Spark, Delta Lake & AI Pipelines

Reuben Cooley Inc.; Cary (NC); On-site; USD 120,000 - 180,000; Full time

Job summary

Reuben Cooley Inc. seeks a senior data engineer to architect and deliver production-grade pipelines using Python, Scala, PySpark, Spark, and Delta Lake.

You will optimize batch and streaming workloads, ensure robust schemas and data contracts, and own Databricks platforms and governance. The role requires 12–18 years of relevant experience, strong security practices, and deep expertise in AI data paths, LangChain tooling, and model serving.

Qualifications

• Expert-level proficiency in Python, Scala, and PySpark with production-grade pipelines.
• Strong SQL and data modelling, schema design and data contracts.
• Databricks Delta Lake, Unity Catalog, Jobs & Workflows, and cluster management.
• 12–18 years of total experience in data engineering or related disciplines.
• Proven delivery of a medallion/lakehouse architecture at enterprise scale.
• Azure security and governance: Entra ID, RBAC, Key Vault, private endpoints, and PII handling.
• CI/CD and infrastructure as code: Azure DevOps, Terraform, Databricks Asset Bundles, and automated testing of data pipelines.
• Clear technical writing and the ability to present and defend a design to engineers and non-technical stakeholders.

Responsibilities

• Design and ship production-grade data pipelines using Python, Scala, and Spark.
• Optimize large-scale batch and streaming workloads with Delta Lake and Spark technologies.
• Define data contracts, schemas, and lineage to ensure reliable data assets.
• Lead AI/ML data paths, including LLM-based pipelines, retrieval, and prompt engineering.

Skills

Python

Scala

PySpark

SQL

Data Modeling

Schema Design

Delta Lake

Databricks

Unity Catalog

Jobs & Workflows

Spark

Tools

Databricks

Azure OpenAI

OpenAI

Terraform

Azure DevOps

LangChain

LlamaIndex

LangGraph

Model Serving

Job description

Reuben Cooley Inc. seeks a senior data engineer to architect and deliver production-grade pipelines using Python, Scala, PySpark, Spark, and Delta Lake.

You will optimize batch and streaming workloads, ensure robust schemas and data contracts, and own Databricks platforms and governance. The role requires 12–18 years of relevant experience, strong security practices, and deep expertise in AI data paths, LangChain tooling, and model serving.