Agentic Data Engineer

Intellibee Inc; Richmond (VA); On-site; USD 80,000 - 110,000; Full time

Job summary

An innovative firm is seeking a highly skilled Agentic Data Engineer to design and deploy data pipelines that leverage cutting-edge AI technology. This role involves creating robust data flows, training large language models, and collaborating with experts to enhance system performance. The ideal candidate will have a strong foundation in data engineering, experience with big data frameworks like Spark and Databricks, and proficiency in Python programming. Join a forward-thinking team dedicated to solving real-world problems through advanced data solutions and make a significant impact in the field of AI and data science.

Qualifications

• Experience in designing data processes to support agentic systems.
• Strong programming skills in Python and experience with AI/ML frameworks.
• Familiarity with cloud computing skills and GIS spatial data.

Responsibilities

• Design and develop data pipelines for agentic systems.
• Train and fine-tune large language models.
• Collaborate with data scientists and engineers to integrate AI.

Skills

Data Engineering Fundamentals

Big Data Frameworks (Spark, Databricks)

Training Large Language Models

Graph Databases

Azure Blob Storage

Azure Data Lakes

Azure Machine Learning

Python Programming

GIS Spatial Data

Education

Bachelor's Degree in Computer Science

Master's Degree in AI or Data Science

Tools

Spark

Azure Databricks

Vector Databases

Job description

Agentic Data Engineer, Richmond, VA, United States

Agentic Data Engineer

Resource will need to be in Richmond, VA quarterly.

Agentic Data Engineer to design, develop, and deploy data pipelines that leverage agentic AI to solve real-world problems.

The Virginia Department of Transportation's Information Technology Division is seeking a highly skilled Agentic Data Engineer to design, develop, and deploy data pipelines that leverage agentic AI to solve real-world problems. The ideal candidate will have experience in designing data processes to support agentic systems, ensure data quality, and facilitate interaction between agents and data.

Responsibilities

1. Design and develop data pipelines for agentic systems, creating robust data flows to handle complex interactions between AI agents and data sources.
2. Train and fine-tune large language models.
3. Design and build data architecture, including databases and data lakes, to support various data engineering tasks.
4. Develop and manage Extract, Load, Transform (ELT) processes to ensure data is accurately and efficiently moved from source systems to analytical platforms.
5. Implement data pipelines that facilitate feedback loops, allowing human input to improve system performance in human-in-the-loop systems.
6. Work with vector databases to store and retrieve embeddings efficiently.
7. Collaborate with data scientists and engineers to preprocess data, train models, and integrate AI into applications.
8. Optimize data storage and retrieval for high performance.
9. Perform statistical analysis, identify trends and patterns, and create data formats from multiple sources.

Qualifications

1. Strong fundamentals in data engineering.
2. Experience with big data frameworks like Spark and Databricks.
3. Experience training large language models with structured and unstructured data sets.
4. Understanding of Graph databases.
5. Experience with Azure Blob Storage, Azure Data Lakes, Azure Databricks.
6. Experience implementing Azure Machine Learning, Azure Computer Vision, Azure Video Indexer, Azure OpenAI models, Azure Media Services, Azure AI Search.
7. Knowledge of effective data partitioning criteria and implementation using Spark.
8. Understanding of core machine learning concepts and algorithms.
9. Familiarity with cloud computing skills.
10. Strong programming skills in Python and experience with AI/ML frameworks.
11. Proficiency with vector databases and embedding models for retrieval tasks.
12. Experience integrating with AI agent frameworks.
13. Experience with cloud AI services (Azure AI).
14. Experience working with GIS spatial data for mapping and geolocation tasks.
15. Experience with Department of Transportation data domains, developing AI solutions for data analysis, hypothesis validation, forecasting, and what-if analysis.
16. Bachelor's or master's degree in computer science, AI, Data Science, or a related field.

Skill Matrix

1. At least 1 year of understanding big data technologies.
2. At least 1 year of experience developing ETL and ELT pipelines.
3. At least 1 year of experience with Spark, GraphDB, Azure Databricks.
4. At least 1 year of expertise in data partitioning.
5. At least 3 years of experience in data conflation.
6. At least 3 years of experience developing Python scripts.
7. At least 2 years of experience training LLMs with structured and unstructured data sets.
8. At least 3 years of experience working with GIS spatial data.