Senior Data Engineer — Databricks/Azure Data Pipelines

independencepetgroup; Chicago (IL); On-site; USD 120,000 - 180,000; Full time

Benefits offered by this job

Healthcare

Parental leave

HSA

FSA

PTO

Retirement plan

Wellness Day off

Group Pet Insurance

Pet-friendly office

Commuter benefits

On-the-job training

EAP

Job summary

Independence Pet Holdings seeks a Senior Data Engineer to design and optimize data pipelines across a modern Delta lake architecture. You will implement robust data quality checks, govern data lineage, and collaborate with cross-functional teams to support product development.

The role requires 7+ years in ELT/ETL, proficiency in Python/SQL/PySpark, and hands-on experience with Databricks and Azure data services. On-site role with a comprehensive benefits package.

Qualifications

• Bachelor's degree in computer science, information technology, or related field.
• 7+ years of experience building data pipelines (ELT/ETL).
• Experience with Databricks, Spark, and Azure data services.
• Strong coding skills in Python, SQL, and PySpark.
• Experience designing CDC and event/stream processing pipelines.
• Proficiency with NoSQL and SQL databases.
• Data modeling, schema design, and performance optimization.
• Familiarity with JSON, Parquet, ORC formats.
• Experience with CI/CD (Azure DevOps, GitLab) and deployment automation.
• Knowledge of data governance, lineage, and cloud security.

Responsibilities

• Design, develop, and deploy scalable data pipelines using Databricks.
• Optimize pipeline performance for cost and efficiency.
• Build robust data models in data lakes for diverse use cases.
• Implement data quality controls and governance practices.
• Integrate Databricks with Azure storage and security services.

Skills

Data engineering

Python

SQL

PySpark

Data modeling

Cloud security

Education

Bachelor's degree in CS/IT

Tools

Databricks

Spark

Azure Data Lake Storage

Azure

Azure DevOps

GitLab

CI/CD

CDC

Job description

Independence Pet Holdings seeks a Senior Data Engineer to design and optimize data pipelines across a modern Delta lake architecture. You will implement robust data quality checks, govern data lineage, and collaborate with cross-functional teams to support product development.

The role requires 7+ years in ELT/ETL, proficiency in Python/SQL/PySpark, and hands-on experience with Databricks and Azure data services. On-site role with a comprehensive benefits package.