Data Engineer: AI-Driven Pipelines & LLM Integration

Insight Global (United States); Charlotte (NC); On-site; USD 120,000 - 180,000; Full time

Job summary

Insight Global is seeking a Data Engineer to build scalable AI-enabled services that connect enterprise data with large language models. The role blends MCP/API development (60%) with data engineering (40%), focusing on model context protocols, APIs, prompts, and cloud-based data pipelines.

The ideal candidate has 5+ years of hands-on data engineering, 3+ years AWS, Python, PySpark, SQL, Kafka, and experience with AWS Glue, S3, Athena, Redshift.

Qualifications

• 5+ years of hands-on data engineering experience.
• 3+ years of hands-on AWS experience.
• Strong understanding of cloud architecture and concepts.
• Hands-on experience with AWS Glue, S3, Athena, and Redshift.
• Experience with APIs and Model Context Protocols (MCPs).
• Python development experience and using PySpark.
• SQL skills.
• Apache Kafka experience.
• Experience building end-to-end data pipelines.
• Strong knowledge of data domains, data modeling, and data analysis.
• Experience designing data ingestion, transformation, and delivery workflows.
• Ability to clearly explain technical decisions through prior project examples.
• Exposure to GenAI solutions using AWS Bedrock, LLMs, tool calling, and agent-based architectures.
• Experience with Claude or Codex and prompt engineering.
• Experience working with large language models (LLMs).
• Experience with Java-based REST APIs, preferably using Spring Boot.
• FastAPI experience.
• Experience with data warehouse architecture.
• Financial services industry experience.
• AWS Step Functions
• Lambda
• Lake Formation
• IAM
• CloudWatch
• Docker
• Git/Jenkins
• RAG
• AI agent development

Responsibilities

• Build scalable AI-enabled services that connect enterprise data with large language models.
• Develop MCPs, APIs, prompts, and cloud-based data pipelines.
• Own data ingestion, transformation, and delivery workflows.
• Collaborate on end-to-end data pipelines and AI integrations.
• Work with GenAI solutions and LLMs in production environments.

Skills

Python

SQL

Pyspark

APIs

LLM integration

Prompt engineering

Data modeling

Data analysis

Data ingestion/transformation/delivery

Java REST APIs

Spring Boot

FastAPI

Data pipelines

Tools

AWS

AWS Glue

S3

Athena

Redshift

Kafka

Docker

Git/Jenkins

Lambda

Step Functions

Lake Formation

IAM

CloudWatch

Spring Boot

FastAPI

Job description

Insight Global is seeking a Data Engineer to build scalable AI-enabled services that connect enterprise data with large language models. The role blends MCP/API development (60%) with data engineering (40%), focusing on model context protocols, APIs, prompts, and cloud-based data pipelines.

The ideal candidate has 5+ years of hands-on data engineering, 3+ years AWS, Python, PySpark, SQL, Kafka, and experience with AWS Glue, S3, Athena, Redshift.