AI-Driven Data Engineer: Scalable Pipelines on Azure

Guidehouse; Fairview Heights (IL); On-site; USD 89,000 - 148,000; Full time

Benefits offered by this job

Medical Insurance

401(k) Retirement Plan

Paid Holidays

Parental Leave

Tuition Reimbursement

Job summary

Guidehouse seeks a Data Engineer with AI experience to support building, optimizing, and maintaining data pipelines, AI solutions, and analytics platforms. You will work with modern data engineering tools, CI/CD, and version control, collaborating with cross‑functional teams to deliver scalable data solutions.

Qualifications include Java, Python, SQL, PySpark, Databricks, and experience with CI/CD and Agile environments.

Qualifications

• Bachelor’s degree required.
• 5 years in data engineering or related roles.
• Experience with Java, Python, SQL, PySpark and API development.
• Experience with CI/CD tools and Git.
• Experience with relational and NoSQL databases (MySQL, PostgreSQL, SQL Server, MongoDB).
• Experience with Databricks.
• Experience with agile development.

Responsibilities

• Design, develop, test and maintain data pipelines and ETL processes.
• AI experience with prompt engineering, machine learning, agentic AI and RAG concepts
• Build and optimize data architectures and models for analytics and reporting.
• Implement CI/CD pipelines for data workflows using Git.
• Develop and deploy cloud-based solutions using Azure Functions and containerization (Docker, Kubernetes).
• Collaborate with DBAs and application developers to design data models and APIs.
• Ensure data quality via validation, monitoring, and logging.
• Troubleshoot production issues to ensure high availability.
• Document data flows and solutions for knowledge sharing and compliance.
• Stay current with emerging technologies in data engineering, cloud, and DevOps.

Skills

Java

Python

SQL

PySpark

APIs

CI/CD

Kubernetes

Docker

Databricks

Agile

Education

Bachelor's degree

Tools

Git

GitHub

Databricks

Kubernetes

Docker

Azure Functions

Job description

Guidehouse seeks a Data Engineer with AI experience to support building, optimizing, and maintaining data pipelines, AI solutions, and analytics platforms. You will work with modern data engineering tools, CI/CD, and version control, collaborating with cross‑functional teams to deliver scalable data solutions.

Qualifications include Java, Python, SQL, PySpark, Databricks, and experience with CI/CD and Agile environments.