Senior Data Engineer - Databricks & PySpark on Azure

AspenView Technology Partners; United States; Hybrid; USD 120,000 - 160,000; Full time

Benefits offered by this job

Competitive base salary

Flexible work model: hybrid, remote, 1

Growth opportunities and leadership

Inclusive culture

Job summary

AspenView Technology Partners in the United States is seeking a Senior Data Engineer to design, develop, and maintain scalable data pipelines using Databricks, Unity Catalog, and Azure Data Factory to support enterprise data modernization.

You will collaborate with architects and stakeholders to deliver secure data governance, Delta Lake architectures, and cloud analytics solutions on Azure, with a flexible hybrid/remote work model.

Qualifications

• 4-7+ years of experience in Data Engineering or Big Data development.
• Strong hands-on experience with Azure Databricks.
• Proficiency in PySpark, Spark SQL, and distributed data processing.
• Experience implementing and managing Unity Catalog.
• Strong experience with Azure Data Factory (ADF).
• Experience working with Delta Lake and modern Lakehouse architectures.
• Solid understanding of data modeling, ETL/ELT, and data warehousing concepts.
• Experience with Azure cloud services and ecosystem.
• Proficiency with Git, version control, and CI/CD processes.
• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management abilities.

Responsibilities

• Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory and Databricks.
• Build and maintain scalable data ingestion frameworks for structured and unstructured data sources.
• Develop data transformation processes using PySpark, Spark SQL, and Databricks notebooks.
• Implement and manage Unity Catalog for enterprise data governance, security, and access controls.
• Create and optimize Delta Lake architectures for analytics and reporting workloads.
• Monitor pipeline performance and troubleshoot production issues.
• Collaborate with data architects to implement enterprise data model and best practices.
• Ensure data quality, integrity, security and compliance standards are met.
• Support CI/CD deployment processes and DevOps practices for data engineering solutions.
• Work closely with business and analytics teams to understand requirements and deliver scalable solutions.

Skills

Databricks expertise

Azure Databricks

PySpark

Spark SQL

Delta Lake

Azure cloud

CI/CD

Git

Tools

Databricks

Azure Data Factory

Unity Catalog

Spark SQL

Delta Lake

Git

Terraform

ARM Templates

Job description

AspenView Technology Partners in the United States is seeking a Senior Data Engineer to design, develop, and maintain scalable data pipelines using Databricks, Unity Catalog, and Azure Data Factory to support enterprise data modernization.

You will collaborate with architects and stakeholders to deliver secure data governance, Delta Lake architectures, and cloud analytics solutions on Azure, with a flexible hybrid/remote work model.