Agentic Data Engineer

TalentBurst; Richmond (VA); Remote; USD 90,000 - 130,000; Full time

Job summary

An innovative company is seeking a highly skilled Agentic Data Engineer to create and manage data pipelines that leverage cutting-edge AI technologies. In this role, you will design robust data architectures, develop ELT processes, and work closely with data scientists to enhance AI systems. This position offers the opportunity to work on exciting projects involving large language models and complex data interactions. If you are passionate about data engineering and AI, and eager to solve real-world problems, this role is perfect for you.

Qualifications

• Proven experience in designing data processes for agentic systems.
• Strong understanding of machine learning concepts and algorithms.

Responsibilities

• Design and develop data pipelines for agentic systems.
• Collaborate with data scientists to integrate AI into applications.

Skills

Data Engineering Fundamentals

Big Data Frameworks (Spark, Databricks)

Training LLMs

Graph Databases

Azure Blob Storage

Azure Data Lakes

Azure Machine Learning

Python Programming

AI/ML Frameworks

GIS Spatial Data

Education

Bachelor's Degree in Computer Science

Master's Degree in AI or Data Science

Tools

Azure Databricks

Azure Computer Vision

Azure Video Indexer

Azure OpenAI Models

Azure Media Services

Azure AI Search

Vector Databases

Job description

Role: Agentic Data Engineer
Duration: 7+ months
Location: Richmond, VA
Details: Remote

Seeking a highly skilled Agentic Data Engineer to design, develop, and deploy data pipelines that leverage agentic AI to solve real-world problems. The ideal candidate will have experience in designing data processes to support agentic systems, ensure data quality, and facilitate interaction between agents and data.

Responsibilities:

1. Design and develop data pipelines for agentic systems, creating robust data flows to manage complex interactions between AI agents and data sources.
2. Train and fine-tune large language models.
3. Design and build data architecture, including databases and data lakes, to support various data engineering tasks.
4. Develop and manage Extract, Load, Transform (ELT) processes to ensure accurate and efficient data movement from sources to analytical platforms.
5. Implement data pipelines that enable feedback loops, incorporating human input to improve system performance in human-in-the-loop systems.
6. Work with vector databases to store and retrieve embeddings efficiently.
7. Collaborate with data scientists and engineers to preprocess data, train models, and integrate AI into applications.
8. Optimize data storage and retrieval for high performance.
9. Perform statistical analysis to identify trends and patterns, creating data formats from multiple sources.

Qualifications:

1. Strong fundamentals in data engineering.
2. Experience with big data frameworks like Spark or Databricks.
3. Experience training LLMs with structured and unstructured datasets.
4. Understanding of Graph Databases.
5. Experience with Azure Blob Storage, Azure Data Lakes, Azure Databricks.
6. Experience implementing Azure Machine Learning, Azure Computer Vision, Azure Video Indexer, Azure OpenAI models, Azure Media Services, and Azure AI Search.
7. Ability to determine effective data partitioning criteria and implement partition schemes using Spark.
8. Understanding of core machine learning concepts and algorithms.
9. Familiarity with cloud computing skills.
10. Strong programming skills in Python and experience with AI/ML frameworks.
11. Proficiency with vector databases and embedding models for retrieval tasks.
12. Expertise in integrating with AI agent frameworks.
13. Experience with cloud AI services, particularly Azure AI.
14. Experience working with GIS spatial data, including mapping and geolocation tasks.
15. Experience developing AI solutions for Department of Transportation data domains, including data modeling, correlation, hypothesis validation, forecasting, and what-if analysis.
16. Bachelor's or master's degree in computer science, AI, Data Science, or a related field.