Senior Data Engineer: Scalable Databricks Pipelines on Azure

Independence Pet Holdings; Virginia (MN); On-site; USD 120,000 - 160,000; Full time

Benefits offered by this job

Healthcare

Parental leave

Paid time off

Commuter benefits

Pet-friendly office

Retirement plan

Wellness day off

Volunteer time

Training & development

EAP

Job summary

Independence Pet Holdings is seeking a Senior Data Engineer to join the data team. The role focuses on building and optimizing data pipelines, ensuring data quality, and maintaining data models across Databricks and Azure data services.

You will work with Python, SQL, and PySpark to deliver reliable data flows supporting product development across North America. A strong emphasis is placed on governance, security, and scalable architecture.

Qualifications

• Bachelor's degree in Computer Science or related field required.
• 7+ years building data pipelines with ELT/ETL.
• Experience with Databricks, Spark, and Azure data services.
• Strong coding in Python, SQL and PySpark.
• config-driven pipelines with CDC and stream processing.
• Proficiency with NoSQL and SQL databases.
• Data modeling, schema design, and performance optimization.
• Experience with data formats like JSON, Parquet, ORC.
• CI/CD for data workflows (Azure DevOps, GitLab).
• Data governance, lineage, and cloud security knowledge.

Responsibilities

• Design, develop, and deploy scalable data pipelines using Databricks.
• Optimize pipeline performance for cost and efficiency.
• Design data models in a data lake with data integrity and accessibility.
• Implement robust data quality and governance checks across pipelines.
• Integrate Databricks with Azure services for storage, compute, and security.

Skills

ELT/ETL pipelines

Data governance

Data modeling

Cloud security

CI/CD pipelines

Python scripting

Education

Bachelor's degree in Computer Science or related field

Tools

Databricks

Azure Data Lake Storage

CI/CD (Azure DevOps, GitLab)

NoSQL databases

SQL

Python

PySpark

Parquet/JSON/ORC

Spark

Job description

Independence Pet Holdings is seeking a Senior Data Engineer to join the data team. The role focuses on building and optimizing data pipelines, ensuring data quality, and maintaining data models across Databricks and Azure data services.

You will work with Python, SQL, and PySpark to deliver reliable data flows supporting product development across North America. A strong emphasis is placed on governance, security, and scalable architecture.