Agentic Data Engineer,Richmond, VA,United States

Intellibee; Richmond (VA); On-site; USD 80,000 - 110,000; Full time

Job summary

An established industry player is seeking a skilled Agentic Data Engineer to design and develop innovative data pipelines leveraging cutting-edge AI technology. This role involves creating robust data architectures and optimizing data flows to enhance interactions between AI agents and data sources. You will collaborate with talented data scientists to preprocess data, train models, and integrate AI into applications. If you have a passion for data engineering and AI, this is a fantastic opportunity to make a significant impact in a dynamic environment, where your contributions will drive real-world solutions and innovations.

Qualifications

• Experience training LLMs with structured and unstructured datasets.
• Understanding of Graph Databases and cloud computing skills.
• Expertise in GIS spatial data and AI integration.

Responsibilities

• Design and develop data pipelines for agentic systems.
• Collaborate with data scientists to preprocess data and train models.
• Implement data pipelines for human-in-the-loop systems.

Skills

Python

Data Engineering

Machine Learning

Azure Databricks

Graph Databases

ETL/ELT Processes

Statistical Analysis

GIS Spatial Data

Big Data Technologies

Cloud Computing

Education

Bachelor's in Computer Science

Master's in Data Science

Tools

Azure Blob Storage

Azure Data Lakes

Spark

Vector Databases

Job description

Agentic Data Engineer, Richmond, VA, United States

Agentic Data Engineer

Resource will need to be in Richmond, VA quarterly.

Agentic Data Engineer to design, develop, and deploy data pipelines that leverage agentic AI to solve real-world problems.

The Virginia Department of Transportation's Information Technology Division is seeking a highly skilled Agentic Data Engineer to design, develop, and deploy data pipelines that leverage agentic AI to solve real-world problems. The ideal candidate will have experience in designing data processes to support agentic systems, ensure data quality, and facilitate interaction between agents and data.

Responsibilities:

1. Designing and developing data pipelines for agentic systems, developing robust data flows to handle complex interactions between AI agents and data sources.
2. Ability to train and fine-tune large language models.
3. Design and build the data architecture, including databases and data lakes, to support various data engineering tasks.
4. Develop and manage Extract, Load, Transform (ELT) processes to ensure data is accurately and efficiently moved from source systems to analytical platforms used in data science.
5. Implement data pipelines that facilitate feedback loops, allowing human input to improve system performance in human-in-the-loop systems.
6. Work with vector databases to store and retrieve embeddings efficiently.
7. Collaborate with data scientists and engineers to preprocess data, train models, and integrate AI into applications.
8. Optimize data storage and retrieval for high performance.
9. Perform statistical analysis to identify trends and patterns, creating data formats from multiple sources.

Qualifications:

1. Experience training LLMs with structured and unstructured data sets.
2. Understanding of Graph Databases.
3. Experience with Azure Blob Storage, Azure Data Lakes, and Azure Databricks.
4. Utilize data storage systems like Spark to implement partition schemes.
5. Understanding of core machine learning concepts and algorithms.
6. Familiarity with cloud computing skills.
7. Strong programming skills in Python and experience with AI/ML frameworks.
8. Proficiency in vector databases and embedding models for retrieval tasks.
9. Expertise in integrating with AI agent frameworks.
10. Experience with cloud AI services (Azure AI).
11. Experience with GIS spatial data to create markers on maps (latitude, longitude, nearest topology of roads, geo-locating between datasets, correlation, etc.).
12. Experience with Department of Transportation data domains, developing an AI composite agentic solution to identify and analyze data models, connect and correlate information to validate hypotheses, forecast, predict, recommend potential strategies, and conduct what-if analysis.
13. Bachelor's or master's degree in computer science, AI, Data Science, or a related field.

Skill Matrix:

1. At least 1 year understanding of big data technologies.
2. At least 1 year experience developing ETL and ELT pipelines.
3. At least 1 year experience with Spark, GraphDB, Azure Databricks.
4. At least 1 year expertise in data partitioning.
5. At least 3 years experience in data conflation.
6. At least 2 years experience training LLMs with structured and unstructured data sets.
7. At least 3 years experience with GIS spatial data.