Senior Data Engineer Databricks, PySpark

AspenView Technology Partners; United States; Hybrid; USD 120,000 - 160,000; Full time

Benefits offered by this job

Competitive base salary

Flexible work model: hybrid, remote, 1

Growth opportunities and leadership

Inclusive culture

Job summary

AspenView Technology Partners in the United States is seeking a Senior Data Engineer to design, develop, and maintain scalable data pipelines using Databricks, Unity Catalog, and Azure Data Factory to support enterprise data modernization.

You will collaborate with architects and stakeholders to deliver secure data governance, Delta Lake architectures, and cloud analytics solutions on Azure, with a flexible hybrid/remote work model.

Qualifications

• 4-7+ years of experience in Data Engineering or Big Data development.
• Strong hands-on experience with Azure Databricks.
• Proficiency in PySpark, Spark SQL, and distributed data processing.
• Experience implementing and managing Unity Catalog.
• Strong experience with Azure Data Factory (ADF).
• Experience working with Delta Lake and modern Lakehouse architectures.
• Solid understanding of data modeling, ETL/ELT, and data warehousing concepts.
• Experience with Azure cloud services and ecosystem.
• Proficiency with Git, version control, and CI/CD processes.
• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management abilities.

Responsibilities

• Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory and Databricks.
• Build and maintain scalable data ingestion frameworks for structured and unstructured data sources.
• Develop data transformation processes using PySpark, Spark SQL, and Databricks notebooks.
• Implement and manage Unity Catalog for enterprise data governance, security, and access controls.
• Create and optimize Delta Lake architectures for analytics and reporting workloads.
• Monitor pipeline performance and troubleshoot production issues.
• Collaborate with data architects to implement enterprise data model and best practices.
• Ensure data quality, integrity, security and compliance standards are met.
• Support CI/CD deployment processes and DevOps practices for data engineering solutions.
• Work closely with business and analytics teams to understand requirements and deliver scalable solutions.

Skills

Databricks expertise

Azure Databricks

PySpark

Spark SQL

Delta Lake

Azure cloud

CI/CD

Git

Tools

Databricks

Azure Data Factory

Unity Catalog

Spark SQL

Delta Lake

Git

Terraform

ARM Templates

Job description

Build the Future with AspenView Technology Partners

At AspenView, we are passionate about transforming the way organizations approach technology. We specialize in creating high-performing, nearshore IT teams to help North American clients innovate faster and more efficiently.

As we continue to grow, we’re looking for exceptional people to join our team and help drive impactful change across industries.

Why Join AspenView?

At AspenView, we’re more than a nearshore IT partner-we’re a people-first, purpose-driven company that believes great culture drives great outcomes. We’re passionate about connecting talent and technology to deliver measurable value for clients—and meaningful career paths for our people.

Here’s what you can expect:

• Competitive base
• Flexible work model: hybrid, remote, or in-office
• Real growth opportunities and leadership visibility
• Inclusive, respectful culture that blends U.S. innovation with Colombian heart
• A company that listens, invests in you, and celebrates wins together

We are looking for a skilled Senior Data Engineer with hands-on experience in Databricks, Unity Catalog, and Azure Data Factory (ADF) to support a large-scale enterprise data modernization initiative. The ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines, enabling secure data governance, and supporting cloud-based analytics solutions in Azure.

What you will do:

This role will collaborate closely with architects, business stakeholders, and cross-functional engineering teams to build reliable, high performance data platforms and accelerate data-driven decision-making.

AI Application Development & Orchestration

• Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory and Databricks.
• Build and maintain scalable data ingestion frameworks for structured and unstructured data sources.
• Develop data transformation processes using PySpark, Spark SQL, and Databricks notebooks.
• Implement and manage Unity Catalog for enterprise data governance, security, and access controls.
• Create and optimize Delta Lake architectures for analytics and reporting workloads.
• Monitor pipeline performance and troubleshoot production issues.
• Collaborate with data architects to implement enterprise data model and best practices.
• Ensure data quality, integrity, security and compliance standards are met.
• Support CI/CD deployment processes and DevOps practices for data engineering solutions.
• Work closely with business and analytics teams to understand requirements and deliver scalable solutions.

What you bring: Experience:

• 4-7+ years of experience in Data Engineering or Big Data development.
• Strong hands-on experience with Azure Databricks.
• Proficiency in PySpark, Spark SQL, and distributed data processing.
• Experience implementing and managing Unity Catalog.
• Strong experience with Azure Data Factory (ADF).
• Experience working with Delta Lake and modern Lakehouse architectures.
• Solid understanding of data modeling, ETL/ELT, and data warehousing concepts.
• Experience with Azure cloud services and ecosystem.
• Proficiency with Git, version control, and CI/CD processes.
• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management abilities.

Preferred Qualifications:

• Experience with Azure Synapse Analytics.
• Familiarity with Collibra or other data governance platforms.
• Experience with Infrastructure as Code (Terraform, ARM Templates).
• Databricks Certification (Data Engineer Associate/Professional).
• Microsoft Azure Data Engineer Associate Certification

Nice to have:

Experience supporting enterprise-scale data migration projects.

Knowledge of healthcare, retail, or consumer products data environments.

Exposure to real-time data processing and streaming technologies.

Visa Sponsorship

AspenView does not sponsor employment visas for this role. Applicants must be currently authorized to work in the United States on a permanent basis without the need for visa sponsorship now or in the future.

Equal Opportunity Employer

AspenView is proud to be an equal opportunity employer. We believe in creating an environment where all employees feel welcome, valued, and empowered to succeed. We celebrate diversity and strive to build a culture of inclusion where all individuals, regardless of their race, color, gender, gender identity or expression, sexual orientation, disability, age, or any other characteristic, can thrive. We encourage applicants from all walks of life to join our team and make a lasting impact.