Generative AI Data Engineer: GPT & RAG Pipelines

Machine Learning Technologies, LLC; Chicago (IL); On-site; USD 140,000 - 200,000; Full time

Job summary

Machine Learning Technologies, LLC in Chicago, IL seeks an experienced Generative AI Data Engineer to design scalable data pipelines and enable GPT-based solutions and RAG workflows. You will prepare, process, and index enterprise data to support enterprise AI initiatives.

You will work with Python, SQL, Spark, and cloud platforms to build end-to-end data architectures, embedding generation, vector search, and governance across data platforms.

Qualifications

• Strong experience in Data Engineering and ETL/ELT pipelines.
• Hands-on experience with Python and SQL.
• Experience with Apache Spark / PySpark.
• Knowledge of GPT, LLMs, and Generative AI concepts.
• Hands-on experience with RAG architecture.
• Experience with vector databases and embedding workflows.
• Experience with cloud platforms (AWS, Azure, or GCP).
• Experience with data warehouses and data lakes.
• Understanding of REST APIs and data integration.

Responsibilities

• Design, develop, and maintain scalable data pipelines for structured and unstructured data.
• Build and optimize ETL/ELT workflows for data ingestion, transformation, and processing.
• Develop data pipelines to support GPT, LLM, and Generative AI applications.
• Implement Retrieval-Augmented Generation (RAG) pipelines.
• Prepare, clean, chunk, and transform documents for embedding generation and indexing.
• Integrate vector databases and embedding models with enterprise data platforms.
• Work with OpenAI GPT models and Azure OpenAI services.
• Develop data workflows using Python, SQL, PySpark, and Spark.
• Implement data quality, validation, governance, and monitoring processes.
• Collaborate with AI/ML engineers, data scientists, and application development teams.

Tools

Python

SQL

Apache Spark / PySpark

Pinecone

Weaviate

FAISS

Azure AI Search

REST APIs

Cloud platforms (AWS, Azure, GCP)

Data warehouses / Data lakes

Vector databases

Job description

Machine Learning Technologies, LLC in Chicago, IL seeks an experienced Generative AI Data Engineer to design scalable data pipelines and enable GPT-based solutions and RAG workflows. You will prepare, process, and index enterprise data to support enterprise AI initiatives.

You will work with Python, SQL, Spark, and cloud platforms to build end-to-end data architectures, embedding generation, vector search, and governance across data platforms.