Senior Qa Engineer- ETL Bigquery

66degrees; Pune District, Bengaluru; Hybrid; INR 1,800,000 - 2,400,000; Full time

Job summary

66degrees is seeking a Senior QA Engineer Data to lead QA for complex data pipelines on Google Cloud Platform. You will design multi-layered tests, including integration, end-to-end, and data quality checks across dbt, BigQuery, Dataflow and more.

You will define testing strategies, collaborate with data engineers, and embed QA in CI/CD pipelines, ensuring data accuracy, reliability, and scalable performance.

Qualifications

• Must have 5+ years in QA focusing on data pipelines and data warehouses.
• Strong SQL and data validation query skills.
• Experience with GCP data tools: dbt, BigQuery, Dataflow, Dataproc, Cloud Run, Spanner, AlloyDB, Cloud SQL.
• Hands-on with data-quality frameworks like Great Expectations, Soda Core, and dbt-expectations.
• Familiarity with test environment provisioning and database emulators.
• CI/CD automation experience with Jenkins, GitLab CI, or GitHub Actions.
• Scripting in Python and pytest.

Responsibilities

• Define and maintain testing methodologies for the full GCP data stack including Dataflow, Dataproc, and dbt.
• Develop and execute data quality tests using Great Expectations, Soda Core, and dbt tests.
• Implement integration and regression tests for ETL/ELT pipelines and Cloud Run tests.
• Test Spanner, Cloud SQL, and AlloyDB using emulators or test instances; validate procedures.
• Orchestrate E2E tests via Cloud Composer or Airflow and validate outputs.
• Embed QA in CI/CD pipelines and provision reproducible test environments with IaC.

Skills

5+ years QA experience

SQL proficiency

CI/CD automation

Python

pytest

communication & mentoring

API testing (Postman)

Tools

dbt

BigQuery

Dataflow

Dataproc

Cloud Run

Spanner

AlloyDB

Cloud SQL

Terraform

CI/CD tools

Job description

Overview of Role

As a Senior QA Engineer Data, you will lead the QA effort for complex data engineering pipelines on Google Cloud Platform. You’ll design and implement multi-layered testing strategies—integration, end-to-end, and data quality tests—across tools like dbt, Dataflow, Dataproc, BigQuery, AlloyDB, Cloud SQL, Cloud composer and Cloud Run, etc. Your work ensures the accuracy, reliability, and performance of data systems at scale.

Responsibilities

Testing Strategy & Test Design

• Define and maintain testing methodologies for the full GCP data engineering stack: Dataflow (TestPipeline), Dataproc (spark-testing-base or pytest), Cloud Run container tests, and SQL-based data validation in BigQuery/dbt.
• Develop and execute data quality frameworks using dbt tests (schema, singular, freshness) and external tools like Great Expectations, Soda Core, and Dataplex.

Pipeline & Database Testing

• Implement integration, and regression tests for ETL/ELT pipelines, including container- level and HTTP-triggered tests for Cloud Run.
• Use emulators or dedicated test instances to test Spanner, Cloud SQL, and AlloyDB. Validate stored procedures and database functions with sample data.

End-to-End (E2E) Pipeline Validation

• Orchestrate comprehensive E2E tests via Cloud Composer/Apache Airflow or scripting. Simulate real-world data flows and validate intermediate and final outputs.

CI/CD & Automation

• Embed QA in CI/CD pipelines (e.g., GitLab CI, Jenkins, GitHub Actions), automating test execution at all levels including data quality validations and dbt runs.
• Use IaC tools (Terraform, Deployment Manager) to provision reproducible test environments.

Collaboration & Stakeholder Engagement

• Partner with data engineers and stakeholders to review design for testability.
• Mentor junior QA team members, champion QA best practices, and lead efforts to improve data quality KPIs and test effectiveness.

Monitoring & Observability

• Utilize Cloud Logging, Monitoring, and observability tools (e.g., Elementary Data) to track pipeline health, test results, and identify anomalies.

Required Qualifications

• 5+ years in QA or software testing, including focused on data pipelines/data warehouses
• Proficient in complex SQL and writing data validation queries.
• Strong experience with GCP data tools: dbt, BigQuery, Dataflow, Dataproc, Cloud Run, Spanner, AlloyDB, Cloud SQL.
• Hands-on with data-quality frameworks: Great Expectations, Soda Core, dbt-utils, dbt-expectations.
• Familiarity with database emulators and test environment provisioning.
• CI/CD automation experience (Jenkins, GitLab CI, GitHub Actions).
• Skilled in scripting languages (Python, pytest).
• Excellent communication, mentoring ability, critical thinking, and attention to detail.
• API testing with PI Testing Postman REST APIs

Preferred Qualifications

• GCP Professional Data Engineer certification
• Experience enhancing dbt tests with dbt-utils and dbt-expectations.
• Knowledge of BigQuery data-quality services like Dataplex.
• Proficient with observability tooling (Cloud Logging, Elementary Data).
• Familiarity with data governance, lineage, and compliance validation.
• Exposure enterprise-based data migration project
• Exposure to Datawarehouse modernization