Senior Data Engineer

Independence Pet Group; Chicago, Northern (IL, KY); Hybrid; USD 120,000 - 180,000; Full time

Benefits offered by this job

Medical Insurance

Dental Insurance

Vision Insurance

Parental Leave 12 weeks

HSA/FSA

Retirement Plan

Paid Time Off

Wellness Day Off

Pet-friendly Office

EAP

Job summary

Independence Pet Holdings seeks a Senior Data Engineer to build and optimize data pipelines across a medallion architecture. You will ensure data quality and governance while supporting product development.

You will design scalable data models, implement robust checks, and integrate Databricks with Azure services, enabling secure, reliable data flows across multiple brands.

Qualifications

• Bachelor’s degree in computer science, IT, or related field.
• 7+ years of experience building data pipelines using ELT/ETL.
• Experience with Databricks, Spark, and Azure data services.
• Strong coding skills in Python, SQL, and PySpark.
• Experience with CDC and event/stream processing.
• Proficiency with NoSQL and SQL databases.
• Solid data modeling, schema design, and performance optimization.
• Familiarity with JSON, Parquet, and ORC formats.
• Hands-on CI/CD experience (Azure DevOps, GitLab).
• Knowledge of data governance, lineage, and cloud security.

Responsibilities

• Design, develop, and deploy robust data pipelines using Databricks and Azure.
• Optimize pipeline performance for efficiency and cost effect.
• Model data in the data lake for diverse use cases with integrity.
• Implement data quality checks and validation across pipelines.
• Adhere to data governance policies and best practices.
• Integrate Databricks with Azure storage, compute, and security services.

Skills

Databricks

Spark

Azure Data Services

Python

SQL

PySpark

CDC

Event streaming

NoSQL databases

SQL databases

Data modeling

Schema design

Data governance

CI/CD pipelines

Azure DevOps

GitLab

Data quality checks

Cloud security

Unity Catalog

Troubleshooting

Education

Bachelor’s degree in computer science, Information Technology, or related field

Tools

Databricks

Azure Data Lake Storage

Azure DevOps

GitLab

CI/CD tooling

Job description

Established in 2021, Independence Pet Holdings is a corporate holding company that manages a diverse and broad portfolio of modern pet health brands and services, including insurance, pet education, lost recovery services, and more throughout North America.

We believe pet insurance is more than a financial product and build solutions to simplify the pet parenting journey and help improve the well‑being of pets.

As a leading authority in the pet category, we operate with a full stack of resources, capital, and services to support pet parents.

Our multi‑brand and omni‑channel approach include our own insurance carrier, insurance brands and partner brands.

About US:Clarus, a Doubtless company, is a pioneering pet health technology company on a mission to transform the future of pet healthcare.

We are dynamic entrepreneurs improving pet health by providing trusted and transparent information across the entire pet health landscape.

Join us inshapinga smarter, healthier future for pets everywhere.

As we expand, we are looking for a seasonedSenior Data Engineer, in the data team, who can hit the ground running by building data pipelines across the medallion architecture.

Role Overview:

We are seeking a skilled Data Engineer to join our team. The successful candidate will be responsible for development and optimization of data pipelines, implementing robust data checks, and ensuring the accuracy and integrity of data flows. This role is critical in supporting the product development processes.

Key Responsibilities:

• Design, develop, and deploy robust and scalable data pipelines using Databricks, integrating data from various sources.
• Optimize pipeline performance for greater efficiency and cost-effectiveness.
• Contribute to the design and implementation of robust data models in the data lake environment, ensuring data consistency, integrity and accessibility for various use cases.
• Implement rigorous data quality controls and validation procedures throughout the data pipeline to ensure high accuracy and reliability.
• Comply with the data governance policies and best practices.
• Seamlessly integrate Databricks with Azure cloud services for storage, compute and security, leveraging services like Azure Data Lake Storage.

Qualifications:

• Bachelor’s degree in computer science, Information Technology, or related field.
• At least 7 years of experience in building data pipelines using ELT/ETL.
• Experience working with Databricks, Spark, and Azure data services.
• Strong coding skills with Python, SQL and PySpark.
• Experience designing and maintaining config-driven pipelines with support for Change Data Capture (CDC) and event/stream-based processing.
• Proficiency with NoSQL and SQL databases.
• Strong understanding of data modeling, schema design, and database performance optimization.
• Practical experience working with various file formats, including JSON, Parquet, and ORC.
• Hands‑on experience building and maintaining CI/CD pipelines (Azure DevOps, GitLab) and automating data workflow deployments.
• Solid understanding of data governance, lineage, and cloud security (Unity Catalog, encryption, access control).
• Excellent problem‑solving and analytical skills.
• Experience with data modeling tools and techniques.
• Knowledge of cloud-based data solutions and platforms.

Some offerings are dependent upon the location of where you work and can include the following:

• Comprehensive full medical, dental and vision Insurance
• Basic Life Insurance at no cost to the employee
• Company paid short-term and long-term disability
• 12 weeks of 100% paid Parental Leave
• Health Savings Account (HSA)
• Flexible Spending Accounts (FSA)
• Retirement savings plan
• Personal Paid Time Off
• Paid holidays and company-wide Wellness Day off
• Paid time off to volunteer at nonprofit organizations
• Pet friendly office environment
• Commuter Benefits Group
• Pet Insurance
• On the job training and skills development
• Employee Assistance Program (EAP)

Interview Technology Notice: Please note that phone and video interviews or screenings may be recorded and transcribed using interview technology to support our recruitment process. By continuing with the interview, you consent to this use.

Text Messaging Notice: If you provide a mobile phone number, you may receive job‑related communications via text message. Message and data rates may apply. You may opt out of text communications at any time by replying “STOP.”

Thanks for visiting our careers page!

ASPCA Pet Health Insurance, Embrace, Felix, Figo, Independence American Insurance Company, Pets Best, Pethealth, PetPartners, PetPlace, and Pumpkin are all operated by Independence Pet Holdings (IPH).

At IPH, we welcome and encourage applications from individuals with disabilities.

Accommodations are available upon request for candidates participating in all aspects of the selection process.