Senior Test Architect: Python/PySpark Data Pipelines on Azure

TechDigital Group; United States; Hybrid; USD 120,000 - 150,000; Full time

Job summary

A leading technology firm in the United States is seeking a Test Manager / Test Architect specialized in Azure Databricks, Python, and PySpark. This role involves managing end-to-end data pipelines and collaborating with cross-functional teams to ensure data quality. Candidates should possess over 12 years of data engineering experience, including expertise in ETL processes and Azure services. Proficiency in CI/CD and strong communication skills are essential for this dynamic position.

Qualifications

• 12+ years of experience in data engineering with expertise in Python and PySpark.
• 3+ years of experience working with Azure Databricks and other Azure data services.
• Proven experience in ETL and data migration projects.

Responsibilities

• Manage Testing of end-to-end data pipelines using Azure Databricks, Python, and PySpark.
• Collaborate with cross-functional teams including data scientists and analysts.
• Ensure data quality, integrity, and governance across all migration processes.

Skills

Data engineering

Python

PySpark

ETL

Data migration

Cloud data engineering

CI/CD pipelines

Communication

Problem-solving

Project management

Education

Bachelor's or Master's degree in Computer Science, Engineering, or related field

Tools

Azure Databricks

Azure Data Lake

Azure Synapse

Git

Azure DevOps

Job description

A leading technology firm in the United States is seeking a Test Manager / Test Architect specialized in Azure Databricks, Python, and PySpark. This role involves managing end-to-end data pipelines and collaborating with cross-functional teams to ensure data quality. Candidates should possess over 12 years of data engineering experience, including expertise in ETL processes and Azure services. Proficiency in CI/CD and strong communication skills are essential for this dynamic position.