Senior Data Engineer QA

Atyeti; Mumbai; Hybrid; INR 2,600,000 - 3,800,000; Full time

Job summary

Atyeti is seeking a Senior QA Data Engineer in Mumbai to lead data-quality validation across Azure Data Factory, Databricks, and Synapse. You will design scalable automation and oversee end-to-end testing of ETL/ELT processes, BI reporting, and data pipelines.

The role emphasizes collaboration with cross-functional teams, implementing robust validation, and ensuring reliable data workflows in a fast-paced environment. Offshore/global projects may be involved.

Qualifications

• Data quality & validation on Azure data platforms.
• ADF & Databricks validation across complex workflows.
• ETL/ELT testing: ingestion, transformations, reconciliation.
• Lineage, auditing, and monitoring using observability tools.

Responsibilities

• Plan, design, and execute comprehensive testing strategies for new features, enhancements, and system changes.
• Create, maintain, and execute manual and automated test cases ensuring thorough coverage.
• Champion automation best practices by designing scalable, maintainable solutions supporting continuous delivery.
• Validate system quality across the SDLC including performance and reliability.
• Identify, analyse, and troubleshoot defects with root-cause analysis and cross-functional collaboration.
• Produce clear, detailed defect reports to support remediation.
• Share progress, risks, and quality insights with technical and non-technical stakeholders.

Skills

Azure Data Factory

Databricks

Synapse

SQL

Python

PySpark

Pandas

C#

REST APIs

DAX

Power BI

Power Apps

Dataverse

Azure DevOps

GitHub Actions

Great Expectations

DataDiffPy

Tools

PyTest

Postman

Great Expectations

DataDiffPy

Job description

Job Description - Senior QA Data Engineer

Key Responsibilities:

• Testing Strategy: Plan, design, and execute comprehensive testing strategies for new features, enhancements, and system changes.
• Hands-on Testing: Create, maintain, and execute manual and automated test cases ensuring thorough coverage.
• Automation Leadership: Champion automation best practices by designing scalable, maintainable solutions supporting continuous delivery.
• Functional & Non-functional Testing: Validate system quality, performance, and reliability across the SDLC.
• Defect Management: Identify, analyse, and troubleshoot defects with root cause analysis, collaborating with cross-functional teams for resolution.
• Reporting: Produce clear, detailed defect reports to support efficient remediation.
• Stakeholder Communication: Share progress, risks, and quality insights with technical and non-technical stakeholders.

Knowledge & Experience Required:

Data Quality & Validation (Azure Data Platforms)

• ADF & Databricks Validation: Ensure accuracy, reliability, and consistency across complex workflows.
• ETL/ELT Testing: Validate ingestion, transformations, schema, and reconciliation between source and target systems.
• Data Profiling: Identify anomalies, duplicates, and integrity issues.
• Lineage & Monitoring: Validate lineage, auditing, and monitoring using observability tools.

Test Automation for Data Platforms

• Automation Frameworks: Design frameworks using Python (PyTest, PySpark, Pandas) or C#.
• Reusable Scripts: Build scripts for pipelines, transformations, and integrations.
• SQL Integration Tests: Develop integration tests for queries and workflows.

SQL & Database Testing

• Advanced SQL: Validate business rules, transformations, and workflows.
• Source-to-Target Validation: Ensure accuracy post-transformations.
• Database Structures: Validate indexes, constraints, stored procedures, and performance.

Business Intelligence & Reporting Testing

• BI Tools: Validate dashboards, reports, datasets, and models.
• Power BI: Work with DAX queries and tools like DAX Studio.
• Power Platform: Test Power Apps and Dataverse solutions.

CI/CD & Data Pipeline Testing

• Pipeline Integration: Embed validation tests into Azure DevOps/GitHub Actions.
• Deployment Validation: Support deployment validation for ETL pipelines and BI models.

Data Warehouse & Traditional ETL Tools

• Warehouse Testing: Validate architectures, data modelling, and enterprise platforms.
• ETL Tools: Experience with SAP BO Data Services, Informatica, SSIS.

Quality Engineering & Collaboration

• Enterprise QA: Strong focus on automation and quality engineering practices.
• Agile Practices: Active contribution to sprint ceremonies and Three Amigos sessions.
• Mentorship: Mentor QA engineers, conduct code reviews, and promote best practices.
• Collaboration: Partner with BAs and Developers to refine requirements and acceptance criteria.

Essential Skills:

• Cloud & Data Platforms: Azure Data Factory, Databricks, Synapse, SQL.
• Databases & Storage: SQL Server, Delta Lake.
• Programming & Scripting: Python (PySpark, Pandas), SQL, C#, REST APIs, DAX.
• Automation Tools: PyTest, Great Expectations, DataDiffPy, Postman.
• CI/CD & DevOps: Azure DevOps, GitHub Actions.
• Monitoring: Azure Monitor, Log Analytics, Databricks Monitoring.
• BI & Power Platform: Power BI, Power Apps, Dataverse.
• Database Testing: Advanced SQL for validation and performance analysis.
• Collaboration & Soft Skills: Strong teamwork, communication, and Agile collaboration.