Engineer (India Office)

Wesco; Bengaluru; On-site; INR 1,800,000 - 2,600,000; Full time

Job summary

Wesco in Bengaluru seeks a data professional to design and validate end-to-end data pipelines across Azure Data Factory, Databricks, and ADLS, focusing on data quality, automation with Python/PySpark, and CI/CD integration.

The role emphasizes large-scale data validation, dashboards via Power BI/Tableau/Looker, and reliable batch/streaming processes with KPI-driven data health monitoring.

Qualifications

• 4 - 6 years experience in enterprise data modeling and data architecture roles.
• Strong hands-on experience with Azure Data Factory & PySpark in large-scale environments is preferred.
• Experience with Databricks (Delta Lake, Spark, Unity Catalog).
• Advanced SQL and strong understanding of PySpark scripts for data validation.
• Experience integrating major enterprise applications (ERP, CRM, OMS, MDM, AR systems).
• Excellent communication skills across business and technical audiences.

Responsibilities

• Design and implement end-to-end data validation strategies for pipelines built on Azure Data Factory, Databricks (Delta Lake), and ADLS.
• Perform source-to-target reconciliation across ingestion, transformation, and consumption layers.
• Implement robust checks for Data Quality Dimensions.
• Build scalable data testing frameworks using python, pyspark and automation tools like playwrit​e & pyspark based automation testing.
• Integrate automated data tests into CI/CD pipelines (Azure DevOps / GitHub Actions).
• Validate Spark transformations, Delta Lake tables, and streaming pipelines.
• Perform large-scale data validation using PySpark and SQL.
• Optimize validation logic for high-volume datasets.
• Ensure correctness of Batch/Streaming, CDC, and SCD processes.
• Validate data movement, orchestration workflows, and failure handling in Azure Data Factory & Azure Databricks.
• Define and implement data quality SLAs and KPIs.
• Build dashboards to track data health and pipeline reliability.
• Proactively identify anomalies and data drifts; implement alerting for data failures.
• Design, develop, and optimize interactive dashboards and reports using BI tools (Power BI, Tableau, Looker).
• Translate business requirements into technical reporting solutions.

Skills

Azure Data Factory

PySpark

Databricks

Python

CI/CD

Power BI

Tableau

Looker

Delta Lake

Unity Catalog

SQL

Data validation

Tools

Soda

dbt

Playwright

Job description

Job Description

Data professional with experience in building and validating end-to-end data pipelines across Azure Data Factory, Databricks, and ADLS. Skilled in data quality assurance, automation using Python and PySpark, and integrating tests into CI/CD workflows. Adept at large-scale data validation, monitoring data health, and ensuring reliable batch and streaming processes, with additional expertise in developing scalable BI dashboards and reporting solutions.

• Design and implement end-to-end data validation strategies for pipelines built on Azure Data Factory, Databricks (Delta Lake), and ADLS
• Perform source-to-target reconciliation across ingestion, transformation, and consumption layers
• Implement robust checks for Data Quality Dimensions
• Build scalable data testing frameworks using python, pyspark and good knowledge of automation tools like playwright & pyspark based automation testing.
• Integrate automated data tests into CI/CD pipelines (Azure DevOps / GitHub Actions)
• Validate Spark transformations, Delta Lake tables, and streaming pipelines
• Perform large-scale data validation using PySpark and SQL
• Optimize validation logic for high-volume datasets
• Ensure correctness of: Batch and streaming jobs, Incremental loads (CDC pipelines), Slowly changing dimensions (SCD)
• Validate data movement, orchestration workflows, and failure handling in Azure Data factory & Azure Databricks services
• Define and implement data quality SLAs and KPIs
• Build dashboards to track data health and pipeline reliability
• Proactively identify anomalies and data drifts
• Implement alerting mechanisms for data failures
• Design, develop, and optimize interactive dashboards and reports using BI tools (e.g., Power BI, Tableau, Looker).
• Translate business requirements into technical reporting solutions.
• Identify bottlenecks and improve report/query performance.
• Implement best practices for dashboard design and scalability.
• Establish BI standards, frameworks, and best practices.

Job Description

Data professional with experience in building and validating end-to-end data pipelines across Azure Data Factory, Databricks, and ADLS. Skilled in data quality assurance, automation using Python and PySpark, and integrating tests into CI/CD workflows. Adept at large-scale data validation, monitoring data health, and ensuring reliable batch and streaming processes, with additional expertise in developing scalable BI dashboards and reporting solutions.

• Design and implement end-to-end data validation strategies for pipelines built on Azure Data Factory, Databricks (Delta Lake), and ADLS
• Perform source-to-target reconciliation across ingestion, transformation, and consumption layers
• Implement robust checks for Data Quality Dimensions
• Build scalable data testing frameworks using python, pyspark and good knowledge of automation tools like playwright & pyspark based automation testing.
• Integrate automated data tests into CI/CD pipelines (Azure DevOps / GitHub Actions)
• Implement data quality-as-code practices using tools like Soda, dbt etc.
• Validate Spark transformations, Delta Lake tables, and streaming pipelines
• Perform large-scale data validation using PySpark and SQL
• Optimize validation logic for high-volume datasets
• Ensure correctness of: Batch and streaming jobs, Incremental loads (CDC pipelines), Slowly changing dimensions (SCD)
• Validate data movement, orchestration workflows, and failure handling in Azure Data factory & Azure Databricks services
• Define and implement data quality SLAs and KPIs
• Build dashboards to track data health and pipeline reliability
• Proactively identify anomalies and data drifts
• Implement alerting mechanisms for data failures
• Design, develop, and optimize interactive dashboards and reports using BI tools (e.g., Power BI, Tableau, Looker).
• Translate business requirements into technical reporting solutions.
• Identify bottlenecks and improve report/query performance.
• Implement best practices for dashboard design and scalability.
• Establish BI standards, frameworks, and best practices.

Required Qualifications

• 4 - 6 years experience in enterprise data modeling and data architecture roles.
• Strong hands-on experience with Azure Data Factory & Pyspark in large-scale environments is preferred.
• Experience with Databricks (Delta Lake, Spark, Unity Catalog).
• Advanced SQL and strong understanding of pyspark scripts & using pyspark for Data validation.
• Experience integrating major enterprise applications (ERP, CRM, OMS, MDM, AR systems).
• Strong understanding of data governance, data quality, metadata, and lineage.
• Excellent communication skills across business and technical audiences.

Core Technology Stack

• Analytics Platform: Databricks (Delta Lake, Unity Catalog), Microsoft PowerBI
• Languages: SQL, Spark SQL, Python / PySpark
• Governance/Data Quality: Unity Catalog/Informatica DG/DQ

About The Team

At Wesco, we build, connect, power and protect the world. As a leading provider of business-to-business distribution, logistics services and supply chain solutions, we create a world that you can depend on.

Our Company’s greatest asset is our people. Wesco is committed to fostering a workplace where every individual is respected, valued, and empowered to succeed. We promote a culture that is grounded in teamwork and respect. With a workforce of over 20,000 people worldwide, we embrace the unique perspectives each person brings. Through comprehensive benefits and active community engagement, we create an environment where every team member has the opportunity to thrive.

Learn more about Working at Wesco here and apply online today!

Founded in 1922 and headquartered in Pittsburgh, Wesco is a publicly traded (NYSE: WCC) FORTUNE 500® company.

Wesco International, Inc., including its subsidiaries and affiliates (“Wesco”) provides equal employment opportunities to all employees and applicants for employment. Employment decisions are made without regard to race, religion, color, national or ethnic origin, sex, sexual orientation, gender identity or expression, age, disability, or other characteristics protected by law. US applicants only, we are an Equal Opportunity Employer.

Los Angeles Unincorporated County Candidates Only: Qualified applicants with arrest or conviction records will be considered for employment in accordance with the Los Angeles County Fair Chance Ordinance and the California Fair Chance Act.

This posting is for a current, active vacancy intended for immediate hire.