Senior Spark QA Engineer

Remote Jobs; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

Remote Jobs is seeking a Senior Spark QA Engineer to lead functional and performance testing of Apache Spark applications and large-scale data platforms. This remote role requires strong hands-on experience with Spark, distributed systems, and CI/CD practices.

You will design tests for Spark jobs and ETL pipelines, set up clusters on Kubernetes, and mentor the QA team. Proficiency in Python or Java and knowledge of Spark optimization are essential for delivering scalable, reliable data

Qualifications

• 5+ years of QA/testing experience with strong hands-on expertise in Apache Spark.
• Experience in functional and performance testing of distributed systems.
• Strong understanding of Spark architecture and optimization techniques.
• Experience with Kubernetes and cloud-based Spark environments.
• Proficiency in Python or Java.
• Experience with automation frameworks and CI/CD pipelines.

Responsibilities

• Perform manual and automated testing of Spark jobs, Spark SQL queries, and ETL pipelines.
• Execute functional, scalability, and performance testing for Spark workloads.
• Set up and manage Spark clusters on Standalone, YARN, Kubernetes, Mesos, Databricks, EMR, and Dataproc.
• Conduct benchmarking, validation, and performance analysis of Spark applications.
• Identify bottlenecks and troubleshoot distributed system issues.
• Lead QA initiatives and mentor team members.

Skills

Apache Spark

QA testing

Distributed systems

Kubernetes

Python/Java

Tools

CI/CD pipelines

Automation frameworks

Job description

Senior Spark QA Engineer – Functional & Performance Testing

Location: Remote

Experience: 5+ Years

Job Description

We are looking for a Senior Spark QA Engineer with strong expertise in functional and performance testing of Apache Spark applications and distributed data platforms.

Key Responsibilities

• Perform manual and automated testing of Spark jobs, Spark SQL queries, and ETL pipelines.
• Execute functional, scalability, and performance testing for Spark workloads.
• Set up and manage Spark clusters on Standalone, YARN, Kubernetes, Mesos, Databricks, EMR, and Dataproc.
• Conduct benchmarking, validation, and performance analysis of Spark applications.
• Identify bottlenecks and troubleshoot distributed system issues.
• Lead QA initiatives and mentor team members.

Required Skills

• 5+ years of QA/testing experience with strong hands-on expertise in Apache Spark.
• Experience in functional and performance testing of distributed systems.
• Strong understanding of Spark architecture and optimization techniques.
• Experience with Kubernetes and cloud-based Spark environments.
• Proficiency in Python or Java.
• Experience with automation frameworks and CI/CD pipelines.