AI Data Engineer: Vector Embeddings & Scalable Pipelines

Cloud Analytics Technologies, LLC; Atlanta (GA); On-site; USD 110,000 - 150,000; Full time

Job summary

Cloud Analytics Technologies, LLC is seeking an experienced AI Data Engineer to design, build, and maintain scalable data pipelines and infrastructure for AI/ML initiatives. The role emphasizes SQL, Python/Scala, and cloud data platforms with hands-on work on vector embeddings, RAG pipelines, and secure data access.

Collaboration with ML Engineers and Data Scientists is essential to enable model training and evaluation.

Qualifications

• Strong SQL and proficiency in Python or Scala.
• Experience with cloud data platforms.
• Hands-on experience with Apache Airflow, dbt, or similar orchestration tools.
• Understanding of embedding models, vector search, and RAG architectures.

Responsibilities

• Design and develop data pipelines to ingest, clean, and transform structured and unstructured data.
• Build and maintain vector embedding pipelines and integrate vector databases.
• Develop data pipelines and feature stores for ML model training and inference.
• Implement RAG pipelines, including document chunking, embedding refresh, and metadata tagging.
• Collaborate with ML Engineers and Data Scientists to support model training and evaluation.
• Ensure data quality, lineage, governance, and security across AI data pipelines.
• Optimize data pipelines for performance, scalability, and cost efficiency.
• Support MLOps/LLMOps workflows, including dataset versioning and reproducibility.
• Implement data access controls and entitlement logic to ensure secure AI data usage.

Skills

SQL

Python/Scala

Cloud data platforms

Airflow

dbt

Vector databases

RAG architectures

Tools

Airflow

dbt

Job description

Cloud Analytics Technologies, LLC is seeking an experienced AI Data Engineer to design, build, and maintain scalable data pipelines and infrastructure for AI/ML initiatives. The role emphasizes SQL, Python/Scala, and cloud data platforms with hands-on work on vector embeddings, RAG pipelines, and secure data access.

Collaboration with ML Engineers and Data Scientists is essential to enable model training and evaluation.