AI-Driven Data Engineer: Pipelines, Cloud & ML Ops

Guidehouse; Fairview Heights (IL); On-site; USD 89,000 - 148,000; Full time

Benefits offered by this job

Medical, Rx, Dental & Vision Insurance

401(k) Retirement Plan

Parental Leave

Tuition Reimbursement

Job summary

Guidehouse seeks a Data Engineer with AI experience to design, build, and maintain data pipelines and analytics platforms. You will work with SQL, Python, Java, and PySpark to ingest, transform, and load data, while implementing CI/CD and cloud-based solutions on Azure.

Collaboration with DBAs and developers is essential, with a focus on data quality and scalable architectures. The role emphasizes containerization with Docker/Kubernetes, experience in Databricks, and keeping up with modern data

Qualifications

• Bachelor's degree required; 5+ years of data engineering or related experience.

Responsibilities

• Design, develop, test and maintain data pipelines and ETL processes from diverse sources (SQL, R, Python).
• AI experience with prompt engineering, ML, agentic AI, and RAG concepts.
• Build and optimize data architectures and models to support analytics, reporting and operations.
• Implement and manage CI/CD pipelines for data workflows using Git.
• Develop and deploy cloud-based solutions with Azure Functions and containerization (Docker, Kubernetes).
• Collaborate with DBAs and developers to design data models, stored procedures and APIs.
• Ensure data quality through validation, monitoring and logging.
• Troubleshoot production issues to ensure high availability and reliability.
• Document data flows and technical solutions for knowledge sharing and compliance.
• Stay current with emerging technologies in data engineering, cloud and DevOps.

Skills

Java

Python

SQL

PySpark

APIs

CI/CD

Data pipelines

Data modeling

Education

Bachelor's degree

Tools

Git

Docker

Kubernetes

Databricks

Job description

Guidehouse seeks a Data Engineer with AI experience to design, build, and maintain data pipelines and analytics platforms. You will work with SQL, Python, Java, and PySpark to ingest, transform, and load data, while implementing CI/CD and cloud-based solutions on Azure.

Collaboration with DBAs and developers is essential, with a focus on data quality and scalable architectures. The role emphasizes containerization with Docker/Kubernetes, experience in Databricks, and keeping up with modern data