Generative AI Data Engineer for GPT & RAG Pipelines

Quantum Technologies. LLC; Chicago (IL); On-site; USD 120,000 - 190,000; Full time

Job summary

Quantum Technologies. LLC in Chicago, IL seeks an experienced Generative AI Data Engineer to design scalable data pipelines, process enterprise data, and integrate data platforms with GPT and other Generative AI models.

You will build ETL/ELT workflows, implement RAG pipelines, prepare embedding data, and work with vector databases and cloud platforms while collaborating with AI/ML engineers and data scientists to deliver robust, governed data solutions.

Qualifications

• Experience building scalable data pipelines for structured and unstructured data.
• Hands-on Python and SQL development in production data environments.
• Strong knowledge of Spark / PySpark and ETL/ELT processes.
• Familiarity with GPT/LLMs and RAG workflows.
• Experience with vector databases and embedding workflows.
• Experience on cloud platforms: AWS, Azure, or GCP.
• Understanding of data warehouses, data lakes, and REST APIs.

Responsibilities

• Design, develop, and maintain scalable data pipelines for structured and unstructured data.
• Build and optimize ETL/ELT workflows for data ingestion, transformation, and processing.
• Develop data pipelines to support GPT, LLM, and Generative AI applications.
• Implement Retrieval-Augmented Generation (RAG) pipelines.
• Prepare, clean, chunk, and transform documents for embedding generation and indexing.
• Integrate vector databases and embedding models with enterprise data platforms.
• Work with OpenAI GPT models and Azure OpenAI services.
• Develop data workflows using Python, SQL, PySpark, and Spark.
• Implement data quality, validation, governance, and monitoring processes.
• Collaborate with AI/ML engineers, data scientists, and application development teams.

Skills

Data Engineering

ETL/ELT pipelines

Python

SQL

Apache Spark

PySpark

GPT & LLMs

RAG architecture

Vector databases

Embedding models

Semantic search

Cloud platforms

Data warehouses

Data lakes

REST APIs

Tools

Pinecone

Weaviate

FAISS

Azure AI Search

OpenAI GPT

Job description

Quantum Technologies. LLC in Chicago, IL seeks an experienced Generative AI Data Engineer to design scalable data pipelines, process enterprise data, and integrate data platforms with GPT and other Generative AI models.

You will build ETL/ELT workflows, implement RAG pipelines, prepare embedding data, and work with vector databases and cloud platforms while collaborating with AI/ML engineers and data scientists to deliver robust, governed data solutions.