Generative AI Data Engineer: GPT, LLMs & RAG Pipelines

Valiant Technologies LLC; Chicago (IL); Presencial; USD 120.000 - 170.000; Tempo integral

Resumo da vaga

Valiant Technologies LLC is seeking an experienced Generative AI Data Engineer to design, build, and optimize data pipelines that power GPT-based and RAG-enabled applications. You will process enterprise data, prepare embeddings, and integrate vector databases with cloud platforms.

You will collaborate with AI/ML engineers, data scientists, and application teams to ensure data quality, governance, and scalable architectures using Python, SQL, PySpark, and Spark.

Qualificações

• Experience designing scalable data pipelines for structured and unstructured data.
• Proficiency with Python and SQL.
• Hands-on with Spark/PySpark and vector databases.
• Familiar with GPT/LLMs and RAG architectures.

Responsabilidades

• Design, develop, and maintain scalable data pipelines for structured and unstructured data.
• Build ETL/ELT workflows for ingestion, transformation, and processing.
• Develop GPT/LLM Generative AI data pipelines.
• Implement Retrieval-Augmented Generation (RAG) pipelines.
• Prepare, clean, chunk, and transform documents for embedding generation and indexing.
• Integrate vector databases and embedding models with enterprise data platforms.
• Work with OpenAI GPT models and Azure OpenAI services.
• Develop data workflows using Python, SQL, PySpark, and Spark.
• Implement data quality, validation, governance, and monitoring processes.
• Collaborate with AI/ML engineers, data scientists, and application development teams.

Conhecimentos

Data engineering

Python & SQL

Spark / PySpark

GPT/LLM concepts

RAG architecture

Vector databases

Embeddings & semantic search

Cloud platforms

Data warehouses & lakes

API integration

Ferramentas

Pinecone

Weaviate

FAISS

Azure AI Search

Descrição da vaga

Valiant Technologies LLC is seeking an experienced Generative AI Data Engineer to design, build, and optimize data pipelines that power GPT-based and RAG-enabled applications. You will process enterprise data, prepare embeddings, and integrate vector databases with cloud platforms.

You will collaborate with AI/ML engineers, data scientists, and application teams to ensure data quality, governance, and scalable architectures using Python, SQL, PySpark, and Spark.