Senior QA Data Engineer

Atyeti; Mumbai; Hybrid; INR 2,500,000 - 4,500,000; Full time

Job summary

Atyeti is seeking a Senior QA Data Engineer to lead validation of data pipelines in Azure Data Factory and Databricks from Mumbai, with extensive experience in ETL/ELT, data profiling, and data quality checks. The role focuses on building automated data validation and BI testing capabilities within Azure environments.

You'll collaborate with analytics and engineering teams, implement CI/CD data validation in pipelines, and ensure robust data accuracy, lineage, and performance for enterprise data

Qualifications

• Extensive experience validating data pipelines in Azure Data Factory (ADF) and Azure Databricks.

Responsibilities

• Plan, design, and execute comprehensive testing strategies for new features, enhancements, and changes across existing systems.
• Take a hands-on role in creating, maintaining, and executing both manual and automated test cases, ensuring thorough test coverage.
• Champion test automation best practices, designing and delivering scalable and maintainable automation solutions that support continuous delivery.
• Develop and execute functional and non-functional test scenarios across various stages of the software development lifecycle to validate system quality, performance, and reliability.
• Identify, analyse, and troubleshoot critical defects, performing root cause analysis and working closely with development and cross-functional teams to ensure timely resolution.
• Produce clear, detailed, and well-structured defect reports that effectively demonstrate issues and support efficient remediation.
• Communicate testing progress, risks, and quality insights to stakeholders at different levels, tailoring communication to suit technical and non-technical audiences.

Skills

Azure Data Factory

Azure Databricks

Azure SQL

Azure Synapse

SQL Server

Delta Lake

Python

PySpark

Pandas

SQL

C#

REST APIs

DAX

PyTest

Great Expectations

DataDiffPy

Postman

Azure DevOps

GitHub Actions

Azure Monitor

Log Analytics

Databricks Monitoring

Power BI

Power Apps

Dataverse

Advanced SQL

Teamwork

Communication

Tools

Postman

Great Expectations

DataDiffPy

PyTest

Job description

Senior QA Data Engineer - Job Description

(Irrelevant profiles won't be considered)

Job Title:Senior QA Data Engineer
Location:Mumbai
Experience Level:5+ Years

Key Responsibilities

• Plan, design, and execute comprehensive testing strategies for new features, enhancements, and changes across existing systems.
• Take a hands-on role in creating, maintaining, and executing both manual and automated test cases, ensuring thorough test coverage.
• Champion test automation best practices, designing and delivering scalable and maintainable automation solutions that support continuous delivery.
• Develop and execute functional and non-functional test scenarios across various stages of the software development lifecycle to validate system quality, performance, and reliability.
• Identify, analyse, and troubleshoot critical defects, performing root cause analysis and working closely with development and cross-functional teams to ensure timely resolution.
• Produce clear, detailed, and well-structured defect reports that effectively demonstrate issues and support efficient remediation.
• Communicate testing progress, risks, and quality insights to stakeholders at different levels, tailoring communication to suit technical and non-technical audiences.

Knowledge & Experience Required:

• Extensive experience validating data pipelines in Azure Data Factory (ADF) and Azure Databricks, ensuring data accuracy, reliability, and consistency across complex data workflows.
• Strong experience testing ETL/ELT processes, including data ingestion, transformations, data movement, and schema validation.
• Perform data completeness, consistency, and reconciliation checks between source and target systems.
• Conduct data profiling to identify anomalies, missing values, duplicates, and data integrity issues.
• Validate data lineage, auditing, and monitoring processes using logging and observability tools.

Test Automation for Data Platforms

• Experience designing and implementing automated data validation frameworks using Python (PyTest, PySpark, Pandas) or C#.
• Build reusable automation scripts to validate data pipelines, transformations, and integrations in Azure environments.
• Develop integration tests for SQL queries, data transformations, and data workflows to ensure reliability and scalability.

SQL & Database Testing

• Advanced proficiency in SQL for validating business rules, transformations, and complex data workflows.
• Perform source-to-target data validation, verifying data accuracy following ETL transformations.
• Validate database structures, indexes, constraints, stored procedures, and query performance for large-scale datasets.
• Business Intelligence & Reporting Testing
• Extensive experience testing BI solutions including Power BI, SAP Business Objects, and Crystal Reports.
• Validate BI dashboards, reports, datasets, and data models against underlying data sources.
• Experience working with Power BI data models, DAX queries, and tools such as DAX Studio.
• Experience testing Power Platform solutions, including Power Apps and Dataverse.

CI/CD & Data Pipeline Testing

• Integrate automated data validation tests into CI/CD pipelines using Azure DevOps or GitHub Actions.
• Implement automated data quality checks within ADF pipelines and Databricks workflows.
• Support deployment validation for ETL pipelines, data models, and transformation workflows.

Data Warehouse & Traditional ETL Tools

• Experience testing data warehouse solutions and traditional ETL tools, including SAP Business Objects Data Services, Informatica, and SSIS.
• Strong understanding of data warehouse architectures, data modelling, and enterprise data platforms.

Quality Engineering & Collaboration

• Extensive experience testing enterprise data platforms, software systems, and data warehouse solutions with a strong focus on test automation and quality engineering practices.
• Solid understanding of the testing pyramid and implementing testing strategies across different layers of the application stack.
• Experience working in Agile development environments, actively contributing to sprint ceremonies and quality practices.
• Proven ability to mentor QA engineers, promote automation best practices, and conduct code reviews.
• Strong collaboration with Business Analysts and Developers to refine requirements, define acceptance criteria, and participate in Three Amigos sessions.
• Excellent analytical, problem-solving, and innovative thinking skills when identifying and resolving complex issues.

Essential skills:

• Cloud & Data Platforms: Azure Data Factory, Azure Databricks, Azure SQL, Azure Synapse Analytics
• Databases & Storage: SQL Server, Delta Lake
• Programming & Scripting: Python (PySpark, Pandas), SQL, C#, REST APIs, DAX
• Automation & Testing Frameworks: PyTest, Great Expectations, DataDiffPy, Postman
• CI/CD & DevOps: Azure DevOps, GitHub Actions
• Monitoring & Observability: Azure Monitor, Log Analytics, Databricks Monitoring
• Business Intelligence & Power Platform: Power BI (dashboards, reports, datasets), Power Apps, Dataverse
• Database Testing: Advanced SQL for data validation, transformation testing, and performance analysis
• Collaboration & Soft Skills: Strong teamwork, communication, and collaboration within cross-functional Agile teams

Desired Experience:

• The ideal candidate will have financial services experience in the private equity, infrastructure & real assets, or private debt space. However, this is not a stringent requirement.

Desired skills:

• Experience working with AI-driven or intelligent data agents, including validating and testing agent-based workflows that interact with enterprise data platforms.
• Exposure to Databricks as a primary data source, including testing data pipelines, queries, and integrations that support AI or agent-based solutions.
• Familiarity with testing AI/ML-enabled systems, including validation of agent behaviour, data retrieval accuracy, and response reliability across data-driven environments.
• Experience working with private markets or financial services platforms, such as eFront or similar private equity / investment management systems.
• Exposure to AI/LLM enabled/accelerated testing and engineering practices.