Engineer India Office - Wesco

Polluxa, Inc.; Karnataka; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Polluxa, Inc. seeks an experienced Data Professional to design and validate end-to-end data pipelines on Azure Data Factory and Databricks.

You will implement data quality checks, build scalable testing frameworks, and integrate automated data tests into CI/CD workflows, ensuring reliability across batch and streaming processes. You will optimize data validation logic, develop BI dashboards, and translate business requirements into robust reporting solutions while collaborating with

Qualifications

• 4 - 6 years experience in enterprise data modeling and data architecture roles.
• Strong hands-on experience with Azure Data Factory & PySpark in large-scale environments.
• Experience with Databricks (Delta Lake, Spark, Unity Catalog).
• Advanced SQL and strong understanding of pyspark scripts & using pyspark for Data validation.
• Experience integrating major enterprise applications (ERP, CRM, OMS, MDM, AR systems).
• Strong understanding of data governance, data quality, metadata, and lineage .
• Excellent communication skills across business and technical audiences.

Responsibilities

• Design and implement end-to-end data validation strategies for pipelines built on Azure Data Factory, Databricks (Delta Lake), and ADLS
• Perform source-to-target reconciliation across ingestion, transformation, and consumption layers
• Implement robust checks for Data Quality Dimensions
• Build scalable data testing frameworks using python, pyspark and good knowledge of automation tools like playwright & pyspark based automation testing.
• Integrate automated data tests into CI/CD pipelines (Azure DevOps / GitHub Actions)
• Implement data quality-as-code practices using tools like Soda, dbt etc.
• Validate Spark transformations, Delta Lake tables, and streaming pipelines
• Perform large-scale data validation using PySpark and SQL
• Optimize validation logic for high-volume datasets
• Ensure correctness of: Batch and streaming jobs, Incremental loads (CDC pipelines), Slowly changing dimensions (SCD)
• Validate data movement, orchestration workflows, and failure handling in Azure Data factory & Azure Databricks services
• Define and implement data quality SLAs and KPIs
• Build dashboards to track data health and pipeline reliability
• Proactively identify anomalies and data drifts
• Implement alerting mechanisms for data failures
• Design, develop, and optimize interactive dashboards and reports using BI tools (e.g., Power BI, Tableau, Looker).
• Translate business requirements into technical reporting solutions.
• Identify bottlenecks and improve report/query performance.
• Implement best practices for dashboard design and scalability.
• Establish BI standards, frameworks, and best practices.

Skills

Data quality assurance

Python

PySpark

CI/CD

Data validation

Data monitoring

BI dashboards

Tools

Azure Data Factory

Databricks

Delta Lake

ADLS

Playwright

Job description

Data professional with experience in building and validating end-to-end data pipelines across Azure Data Factory, Databricks, and ADLS. Skilled in data quality assurance, automation using Python and PySpark, and integrating tests into CI/CD workflows. Adept at large-scale data validation, monitoring data health, and ensuring reliable batch and streaming processes, with additional expertise in developing scalable BI dashboards and reporting solutions.

• Design and implement end-to-end data validation strategies for pipelines built on Azure Data Factory, Databricks (Delta Lake), and ADLS
• Perform source-to-target reconciliation across ingestion, transformation, and consumption layers
• Implement robust checks for Data Quality Dimensions
• Build scalable data testing frameworks using python, pyspark and good knowledge of automation tools like playwright & pyspark based automation testing.
• Integrate automated data tests into CI/CD pipelines (Azure DevOps / GitHub Actions)
• Implement data quality-as-code practices using tools like Soda, dbt etc.
• Validate Spark transformations, Delta Lake tables, and streaming pipelines
• Perform large-scale data validation using PySpark and SQL
• Optimize validation logic for high-volume datasets
• Ensure correctness of: Batch and streaming jobs, Incremental loads (CDC pipelines), Slowly changing dimensions (SCD)
• Validate data movement, orchestration workflows, and failure handling in Azure Data factory & Azure Databricks services
• Define and implement data quality SLAs and KPIs
• Build dashboards to track data health and pipeline reliability
• Proactively identify anomalies and data drifts
• Implement alerting mechanisms for data failures
• Design, develop, and optimize interactive dashboards and reports using BI tools (e.g., Power BI, Tableau, Looker).
• Translate business requirements into technical reporting solutions.
• Identify bottlenecks and improve report/query performance.
• Implement best practices for dashboard design and scalability.
• Establish BI standards, frameworks, and best practices.

Required Qualifications

• 4 - 6 years experience in enterprise data modeling and data architecture roles.
• Strong hands-on experience with Azure Data Factory & Pyspark in large-scale environments is preferred.
• Experience with Databricks (Delta Lake, Spark, Unity Catalog).
• Advanced SQL and strong understanding of pyspark scripts & using pyspark for Data validation.
• Experience integrating major enterprise applications (ERP, CRM, OMS, MDM, AR systems).
• Strong understanding of data governance, data quality, metadata, and lineage .
• Excellent communication skills across business and technical audiences.

Core Technology Stack