AI-Driven Data Engineer (Azure Databricks Pipelines)

Compunnel, Inc.; Chicago (IL); On-site; USD 110,000 - 150,000; Full time

Job summary

Compunnel, Inc. is seeking a hands-on Data Engineer to support AI and data initiatives focused on Azure and Databricks.

The candidate will build scalable data pipelines, process large datasets, and support AI/LLM integrations while working with business stakeholders to deliver high-quality data solutions. Responsibilities include designing and optimizing ingestion, transformation, and modeling processes, ensuring data quality and security, and documenting engineering approaches.

Qualifications

• Strong hands-on experience with Azure Databricks and data engineering.
• Experience in data ingestion, ETL/ELT, and data modeling.
• Experience with large-scale datasets and distributed processing.
• Exposure to AI, Large Language Models (LLMs), or AI integration initiatives.
• Strong SQL and data transformation skills.
• Excellent analytical, problem-solving, and troubleshooting abilities.
• Strong communication skills with business stakeholders, including executive leadership.
• Ability to work independently with minimal supervision.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure and Databricks.
• Build and optimize data ingestion, transformation, and data modeling processes for large datasets.
• Support AI and data engineering initiatives, including LLM integration.
• Collaborate with business stakeholders to understand requirements and deliver data-driven solutions.
• Develop and optimize data workflows for performance, scalability, and reliability.
• Ensure data quality, governance, and security across enterprise data platforms.
• Troubleshoot and resolve data pipeline and platform issues.
• Document technical solutions, data models, and engineering processes.
• Work independently while collaborating with cross-functional business and technical teams.

Skills

Azure Databricks

Data pipelines

ETL/ELT

Data modeling

SQL

Stakeholder comms

Independent work

Tools

Databricks

Job description

Compunnel, Inc. is seeking a hands-on Data Engineer to support AI and data initiatives focused on Azure and Databricks.

The candidate will build scalable data pipelines, process large datasets, and support AI/LLM integrations while working with business stakeholders to deliver high-quality data solutions. Responsibilities include designing and optimizing ingestion, transformation, and modeling processes, ensuring data quality and security, and documenting engineering approaches.