Data Engineer

Compunnel, Inc.; Chicago (IL); On-site; USD 110,000 - 150,000; Full time

Job summary

Compunnel, Inc. is seeking a hands-on Data Engineer to support AI and data initiatives focused on Azure and Databricks.

The candidate will build scalable data pipelines, process large datasets, and support AI/LLM integrations while working with business stakeholders to deliver high-quality data solutions. Responsibilities include designing and optimizing ingestion, transformation, and modeling processes, ensuring data quality and security, and documenting engineering approaches.

Qualifications

• Strong hands-on experience with Azure Databricks and data engineering.
• Experience in data ingestion, ETL/ELT, and data modeling.
• Experience with large-scale datasets and distributed processing.
• Exposure to AI, Large Language Models (LLMs), or AI integration initiatives.
• Strong SQL and data transformation skills.
• Excellent analytical, problem-solving, and troubleshooting abilities.
• Strong communication skills with business stakeholders, including executive leadership.
• Ability to work independently with minimal supervision.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure and Databricks.
• Build and optimize data ingestion, transformation, and data modeling processes for large datasets.
• Support AI and data engineering initiatives, including LLM integration.
• Collaborate with business stakeholders to understand requirements and deliver data-driven solutions.
• Develop and optimize data workflows for performance, scalability, and reliability.
• Ensure data quality, governance, and security across enterprise data platforms.
• Troubleshoot and resolve data pipeline and platform issues.
• Document technical solutions, data models, and engineering processes.
• Work independently while collaborating with cross-functional business and technical teams.

Skills

Azure Databricks

Data pipelines

ETL/ELT

Data modeling

SQL

Stakeholder comms

Independent work

Tools

Databricks

Job description

Job Summary

We are seeking a hands-on Data Engineer to support AI and data initiatives focused on Azure and Databricks. The ideal candidate will have strong experience building scalable data pipelines, processing large datasets, and supporting AI/LLM integrations. This role requires an independent contributor who can work directly with business stakeholders and deliver high-quality data solutions with minimal ramp-up time.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure and Databricks.
• Build and optimize data ingestion, transformation, and data modeling processes for large datasets.
• Support AI and data engineering initiatives, including integration with LLM-based solutions.
• Collaborate with business stakeholders to understand requirements and deliver data-driven solutions.
• Develop and optimize data workflows for performance, scalability, and reliability.
• Ensure data quality, governance, and security across enterprise data platforms.
• Troubleshoot and resolve data pipeline and platform issues.
• Document technical solutions, data models, and engineering processes.
• Work independently while collaborating with cross-functional business and technical teams.

Required Qualifications

• Strong hands-on experience with Azure and Azure Databricks.
• Experience with data engineering, data ingestion, ETL/ELT, and data modeling.
• Experience working with large-scale datasets and distributed data processing.
• Exposure to AI, Large Language Models (LLMs), or AI integration initiatives.
• Strong SQL and data transformation skills.
• Excellent analytical, problem-solving, and troubleshooting abilities.
• Strong communication skills with the ability to work directly with business stakeholders, including executive leadership.
• Ability to work independently with minimal supervision.

Preferred Qualifications

• Experience in the healthcare industry.
• Knowledge of 340B pharmacy programs or pharmacy data.
• Experience with full-stack development.
• Experience supporting AI-driven data platforms and analytics solutions.
• Strong consulting mindset with the ability to quickly adapt to new environments.