Senior AI Data Engineer: Real-Time Pipelines & Vector AI

Visa Hunt; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

Visa Hunt seeks a Senior AI Data Engineer to design and scale AI-focused data pipelines, transforming unstructured data into vectorized formats for LLM consumption. You will own real-time feature pipelines, data quality, lineage, and security across multi-cloud environments.

Applicants bring 10+ years in data engineering, strong Python/SQL, Kafka/Flink/Spark, and experience with vector databases. Collaboration with data science and software teams is essential.

Qualifications

• 10+ years in Data/Backend Engineering with data platforms and pipelines.
• 2+ years supporting AI/ML data pipelines and data prep for ML.
• Expert in Python and SQL; Java/Scala a plus.
• Strong Kafka/Flink/Spark Streaming experience.

Responsibilities

• Design, build, and scale ETL/ELT pipelines for AI workloads (RAG, fine-tuning, batch inference).
• Transform unstructured sources (PDFs, logs, transcripts) to structured/vector formats for LLMs.
• Automate data-to-model lifecycle and keep AI knowledge bases in sync.
• Develop real-time feature pipelines for low-latency AI apps.
• Integrate data platforms with Kafka and event-driven systems for real-time responses.
• Manage and optimize Feature Stores for training vs. production.
• Implement data quality controls and validation for training/inference data.
• Establish data lineage for traceability and compliance.
• Enforce data security, privacy, and governance (PII protection).
• Move and synchronize data across on-prem, cloud, and warehouses.
• Optimize storage/retrieval for Vector Databases for AI searches.
• Collaborate with Data Scientists, ML Engineers, Software Engineers, and stakeholders.

Skills

Python

SQL

Data pipelines

Real-time data streaming

Cross-functional collaboration

Problem solving

Education

Bachelor's degree in CS/SE/IS

Tools

Apache Kafka

Flink

Spark Streaming

Airflow

dbt

Prefect

Kubernetes

Vector Databases

Job description

Visa Hunt seeks a Senior AI Data Engineer to design and scale AI-focused data pipelines, transforming unstructured data into vectorized formats for LLM consumption. You will own real-time feature pipelines, data quality, lineage, and security across multi-cloud environments.

Applicants bring 10+ years in data engineering, strong Python/SQL, Kafka/Flink/Spark, and experience with vector databases. Collaboration with data science and software teams is essential.