Senior Data Engineer: Build Scalable Databricks Pipelines

Independence Pet Holdings; Chicago (IL); On-site; USD 120,000 - 180,000; Full time

Benefits offered by this job

Healthcare

Parental leave

Career advancement

Pet-friendly office

Commuter benefits

Group Pet Insurance

Wellness Day

Volunteer time off

EAP

Job summary

Independence Pet Holdings seeks a Senior Data Engineer to join the data team and build robust data pipelines across a medallion architecture. You will implement data quality controls, data models, and integrate Databricks with Azure for storage and security.

The role emphasizes scalable pipelines, governance, and cost-efficient optimization across data sources to support product development and decision making.

Qualifications

• Bachelor’s degree in computer science, Information Technology, or related field.
• At least 7 years of experience in building data pipelines using ELT/ETL.
• Experience with Databricks, Spark, and Azure data services.
• Strong coding skills with Python, SQL and PySpark.
• Experience designing config-driven pipelines with CDC and event/stream processing.
• Proficiency with NoSQL and SQL databases.
• Strong understanding of data modeling, schema design, and performance optimization.
• Experience with file formats JSON, Parquet, ORC.
• Hands-on experience building CI/CD pipelines (Azure DevOps, GitLab).
• Knowledge of data governance, lineage, and cloud security (Unity Catalog, encryption, access control).
• Excellent problem-solving and analytical skills.
• Experience with data modeling tools and techniques.
• Knowledge of cloud-based data solutions and platforms.

Responsibilities

• Design, develop, and deploy robust and scalable data pipelines using Databricks, integrating data from various sources.
• Optimize pipeline performance for greater efficiency and cost-effectiveness.
• Contribute to the design and implementation of robust data models in the data lake environment, ensuring data consistency, integrity and accessibility for various use cases.
• Implement rigorous data quality controls and validation procedures throughout the data pipeline to ensure high accuracy and reliability. Comply with the data governance policies and best practices.
• Seamlessly integrate Databricks with Azure cloud services for storage, compute and security, leveraging services like Azure Data Lake Storage.

Skills

Python

SQL

PySpark

CDC

NoSQL

Data modeling

Databricks

Education

Bachelor’s degree in Computer Science/IT

Tools

Databricks

Spark

Azure Data Services

CI/CD

Job description

Independence Pet Holdings seeks a Senior Data Engineer to join the data team and build robust data pipelines across a medallion architecture. You will implement data quality controls, data models, and integrate Databricks with Azure for storage and security.

The role emphasizes scalable pipelines, governance, and cost-efficient optimization across data sources to support product development and decision making.