Test Architect

TechDigital Group; United States; Hybrid; USD 120,000 - 150,000; Full time

Job summary

A leading technology firm in the United States is seeking a Test Manager / Test Architect specialized in Azure Databricks, Python, and PySpark. This role involves managing end-to-end data pipelines and collaborating with cross-functional teams to ensure data quality. Candidates should possess over 12 years of data engineering experience, including expertise in ETL processes and Azure services. Proficiency in CI/CD and strong communication skills are essential for this dynamic position.

Qualifications

• 12+ years of experience in data engineering with expertise in Python and PySpark.
• 3+ years of experience working with Azure Databricks and other Azure data services.
• Proven experience in ETL and data migration projects.

Responsibilities

• Manage Testing of end-to-end data pipelines using Azure Databricks, Python, and PySpark.
• Collaborate with cross-functional teams including data scientists and analysts.
• Ensure data quality, integrity, and governance across all migration processes.

Skills

Data engineering

Python

PySpark

ETL

Data migration

Cloud data engineering

CI/CD pipelines

Communication

Problem-solving

Project management

Education

Bachelor's or Master's degree in Computer Science, Engineering, or related field

Tools

Azure Databricks

Azure Data Lake

Azure Synapse

Git

Azure DevOps

Job description

Job Title: Test Manager / Test Architect – Azure Databricks, Python, PySpark – ETL and Migration Specialist

Key Responsibilities:

• Manage Testing of end-to-end data pipelines using Azure Databricks, Python, and PySpark.
• Collaborate with cross-functional teams including data scientists, analysts, and business stakeholders to understand data requirements.
• Ensure data quality, integrity, and governance across all migration and transformation processes.
• Mentor junior engineers and provide technical leadership within the team.
• Develop reusable frameworks and automation tools to streamline migration and data processing tasks.
• Stay updated with the latest trends and best practices in cloud data engineering and migration.

Required Skills & Qualifications:

• Bachelor's or Master's degree in Computer Science, Engineering, or related field.
• 12+ years of experience in data engineering with expertise in Python and PySpark.
• 3+ years of experience working with Azure Databricks and other Azure data services (e.g., Azure Data Lake, Azure Synapse).
• Proven experience in ETL and data migration projects, including strategy, execution, and validation.
• Strong understanding of distributed computing, data modeling, and ETL/ELT processes.
• Experience with CI/CD pipelines and version control tools (e.g., Git, Azure DevOps).
• Excellent problem-solving and communication skills.
• Ability to lead and manage multiple projects simultaneously.

Preferred Qualifications:

• Azure certifications (e.g., Azure Data Engineer Associate).
• Experience with other cloud platforms (AWS, GCP) is a plus.
• Familiarity with Delta Lake, MLflow, and other Databricks ecosystem tools.
• Experience in Agile/Scrum environments.