Data Quality Engineer (Data QE)

Algoleap Technologies; Hyderabad; In loco; INR 1.200.000 - 2.400.000; Tempo pieno

Descrizione del lavoro

Algoleap Technologies Pvt Ltd seeks an experienced Data Quality Engineer to ensure data quality across enterprise platforms. You will validate data pipelines from CDP, GA4, BigQuery, and Blob Storage into Snowflake, and verify reporting in Power BI.

The role covers data governance, metadata management, and comprehensive test automation using SQL, Python, and PySpark in CI/CD environments.

Competenze

• Experience in Data Quality Engineering and Data Warehouse Testing.
• Strong hands-on with MSSQL, PostgreSQL, Snowflake and BigQuery.
• Proven experience validating Source-to-Target from CDP, GA4, BigQuery, Blob Storage, PostgreSQL, and MSSQL into Snowflake.
• Experience validating Power BI reports, dashboards, semantic models, and KPIs.
• Understanding of ETL/ELT data processing, data transformations, and data lineage.
• Experience with Data Governance, Data Quality, Metadata Management, Data Lineage, Data Dictionary, and Data Catalog.
• Familiarity with Data Quality tools such as Soda or equivalents.
• Experience designing, executing, and automatising tests for large data platforms using SQL, Python, and PySpark.
• Ability to integrate automated testing into CI/CD pipelines using Azure DevOps or GitHub Actions.

Mansioni

• End-to-end data validation across ingestion, transformation, storage, and reporting.
• Validate data movement from source systems including CDP, GA4, BigQuery, MSSQL, and Snowflake.
• Perform Source-to-Target validation between sources and Snowflake.
• Assess data accuracy, completeness, and timeliness across datasets.
• Execute data profiling, reconciliation, and schema validation.
• Test Medallion Architecture layers and data quality controls.
• Validate ETL/ELT pipelines and data mappings.
• Ensure data governance policies and metadata management are followed.
• Develop and automate data quality checks and KPIs.

Conoscenze

Data Warehousing

Medallion Architecture

ETL/ELT testing

Data Governance

Data Quality frameworks

Data Lineage

Test automation

PostgreSQL

MSSQL

Snowflake

BigQuery

Power BI

Python

PySpark

Strumenti

Azure Blob Storage

GA4

CDP

Snowflake

BigQuery

Power BI

Azure DevOps

GitHub Actions

Python

PySpark

Descrizione del lavoro

Algoleap Technologies Pvt Ltd | Full time

We are seeking an experienced Data Quality Engineer (DataQE) to ensure the quality, accuracy, reliability, and integrity of dataacross enterprise data platforms. The ideal candidate will have strongexperience in Data Warehousing, Medallion Architecture, ETL/ELT testing,Data Governance, Data Quality frameworks, Data Lineage, and test automation.This role will be responsible for validating data pipelines from multiplesource systems such as CDP, GA4, BigQuery, and Azure Blob Storage into Snowflake,and ensuring accurate reporting in Power BI.

Required Experience

• __years of experience in Data Quality Engineering, Data Warehouse Testing,or Data Validation.
  
• Stronghands-on experience with PostgreSQL, Microsoft SQL Server, Snowflake andBigQuery.
  
• Experiencevalidating enterprise Data Warehouse solutions and Medallion Architectureimplementations.
  
• Provenexperience performing Source-to-Target validation from CDP, GA4, BigQuery,Blob Storage, PostgreSQL, and MSSQL into Snowflake.
  
• Experiencein validating Power BI reports, dashboards, semantic models, and KPIs.
  
• Strongunderstanding of ETL/ELT data processing, data transformations, and datalineage.
  
• Experienceworking with Data Governance, Data Quality, Metadata Management, DataLineage, Data Dictionary, and Data Catalog solutions.
  
• Experiencewith Data Quality tools such as Soda or equivalent.
  
• Experiencedesigning, executing, and automating test cases for large-scale dataplatforms using SQL, Python, and PySpark.
  
• Experienceintegrating automated testing into CI/CD pipelines using Azure DevOps orGitHub Actions.
  

Key Responsibilities

Data Validation & Testing

• Performend-to-end data validation across the data lifecycle, including ingestion,transformation, storage, and reporting.
  
• Validatedata movement from source systems including:
  
• CustomerData Platforms (CDP)
  
• GoogleAnalytics 4 (GA4)
  
• GoogleBigQuery
  
• MSSQL
  
• Snowflake
  
• Performcomprehensive Source-to-Target (S2T) data validation between sourcesystems and Snowflake.
  
• Validatedata accuracy, completeness, consistency, uniqueness, and timelinessacross datasets.
  
• Executedata reconciliation and data profiling activities.
  
• Conductsource-to-target validation across databases, data lakes, and datawarehouses.
  
• Validateschema changes, constraints, indexes, stored procedures, functions,triggers, and views.
  

Medallion Architecture Validation

• Validatedata across Bronze, Silver, and Gold layers within the MedallionArchitecture.
  
• Verifydata transformations, cleansing, aggregates, and business rules betweenlayers.
  
• Ensuredata quality controls are enforced throughout the Medallion framework.
  

ETL/ELT Testing

• Testand validate ETL/ELT pipelines and data transformations.
  
• Validatebusiness mappings, transformation logic, and data lineage.
  
• Verifyincremental and full-load processes.
  
• Performregression, integration, functional, and end-to-end testing of datapipelines.
  
• Identifyand troubleshoot data discrepancies across source and target systems.
  

Snowflake Data Validation

• Validatedata ingestion, transformations, and storage within Snowflake.
  
• Performlarge-scale data validation using SQL and automation scripts.
  
• ValidateSnowflake views, tables, materialized views, stored procedures, and datasharing mechanisms.
  

Power BI Testing

• Validatedatasets, data models, measures, KPIs, and dashboards.
  
• EnsurePower BI reports accurately reflect Snowflake data.
  
• Verifyreport-level calculations, filters, row-level security, and businessmetrics.
  
• Performend-to-end testing from source systems through Snowflake into Power BI.
  

Data Governance & Data Management

• Validateimplementation of Data Governance policies and standards.
  
• Ensurecompliance with enterprise data quality and governance requirements.
  
• Validateand maintain:
  
• DataDictionary
  
• MetadataManagement
  
• BusinessGlossary
  
• DataCatalogs
  
• Collaboratewith Data Governance and Business teams to improve data quality processes.
  

Data Quality Frameworks

• Implementand execute data quality checks and monitoring solutions.
  
• Defineand monitor Data Quality KPIs and metrics.
  
• Developand automate validation rules for:
  
• Accuracy
  
• Consistency
  
• Validity
  
• Timeliness
  
• Supportroot cause analysis and remediation of data quality issues.
  

Test Automation

• Developautomated data validation frameworks and reusable test suites.
  
• Automatesource-to-target reconciliation and regression testing.
  
• Integrateautomated data quality tests into CI/CD pipelines.
  
• Buildautomated validation using SQL, Python, PySpark, and Data Quality tools.
  

Required Technical Skills

Databases & Data Warehousing

• MicrosoftSQL Server (MSSQL)
  
• Snowflake
  
• BigQuery
  
• DataWarehouse Testing
  
• DataModeling Concepts
  
• StarSchema & Snowflake Schema
  
• SlowlyChanging Dimensions (SCD)
  
• Fact& Dimension Validation
  
• CDCValidation
  
• DataMigration Testing
  

SQL Expertise

• AdvancedSQL Query Writing
  
• ComplexJoins
  
• WindowFunctions
  
• Aggregations
  
• QueryOptimization
  
• DataProfiling and Data Analysis
  

Modern Data Platform Validation

• MedallionArchitecture Validation (Bronze, Silver, Gold)