Associate Principal Engineer (India Office)

Wesco; Bengaluru; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

Wesco in Bengaluru is seeking a data professional to design and validate end-to-end data pipelines built on Azure Data Factory, Databricks, and ADLS. You will implement data quality checks, automate tests with Python and PySpark, and integrate them into CI/CD workflows.

You'll validate Spark transformations, Delta Lake tables, and streaming pipelines, build scalable BI dashboards, and ensure data health across batch and streaming processes.

Qualifications

• 8-10 years experience in enterprise data modeling and data architecture.
• Hands-on with Azure Data Factory & PySpark in large-scale environments.
• Experience with Databricks (Delta Lake, Spark, Unity Catalog).
• Advanced SQL and strong understanding of PySpark for data validation.
• Experience integrating major enterprise applications (ERP, CRM, OMS, MDM, AR systems).
• Strong understanding of data governance, data quality, metadata, and lineage.
• Excellent communication skills across business and technical audiences.

Responsibilities

• Design end-to-end data validation strategies for pipelines built on Azure Data Factory, Databricks (Delta Lake), and ADLS.
• Perform source-to-target reconciliation across ingestion, transformation, and consumption layers.
• Implement robust checks for Data Quality Dimensions.
• Build scalable data testing frameworks using Python and PySpark; automation testing with Playwright & PySpark tools.
• Integrate automated data tests into CI/CD pipelines (Azure DevOps / GitHub Actions).
• Implement data quality-as-code practices using Soda, dbt, etc.
• Validate Spark transformations, Delta Lake tables, and streaming pipelines.
• Perform large-scale data validation using PySpark and SQL.
• Optimize validation logic for high-volume datasets.
• Ensure correctness of batch and streaming jobs, CDC, and SCD processes.
• Define and implement data quality SLAs and KPIs; build dashboards to track health.

Skills

Azure Data Factory

Databricks

PySpark

Python

SQL

Power BI

Tools

Azure DevOps

GitHub Actions

Soda

dbt

Delta Lake

Job description

Job Description

Data professional with experience in building and validating end-to-end data pipelines across Azure Data Factory, Databricks, and ADLS. Skilled in data quality assurance, automation using Python and PySpark, and integrating tests into CI/CD workflows. Adept at large-scale data validation, monitoring data health, and ensuring reliable batch and streaming processes, with additional expertise in developing scalable BI dashboards and reporting solutions.

• Design and implement end-to-end data validation strategies for pipelines built on Azure Data Factory, Databricks (Delta Lake), and ADLS
• Perform source-to-target reconciliation across ingestion, transformation, and consumption layers
• Implement robust checks for Data Quality Dimensions
• Build scalable data testing frameworks using python, pyspark and good knowledge of automation tools like playwright & pyspark based automation testing.
• Integrate automated data tests into CI/CD pipelines (Azure DevOps / GitHub Actions)
• Implement data quality-as-code practices using tools like Soda, dbt etc.
• Validate Spark transformations, Delta Lake tables, and streaming pipelines
• Perform large-scale data validation using PySpark and SQL
• Optimize validation logic for high-volume datasets
• Ensure correctness of: Batch and streaming jobs, Incremental loads (CDC pipelines), Slowly changing dimensions (SCD)
• Validate data movement, orchestration workflows, and failure handling in Azure Data factory & Azure Databricks services
• Define and implement data quality SLAs and KPIs
• Build dashboards to track data health and pipeline reliability
• Proactively identify anomalies and data drifts
• Implement alerting mechanisms for data failures
• Design, develop, and optimise interactive dashboards and reports using BI tools (e.g., Power BI, Tableau, Looker).
• Translate business requirements into technical reporting solutions.
• Identify bottlenecks and improve report/query performance.
• Implement best practices for dashboard design and scalability.
• Establish BI standards, frameworks, and best practices.

Required Qualifications

• 8-10 years experience in enterprise data modeling and data architecture roles.
• Strong hands-on experience with Azure Data Factory & Pyspark in large-scale environments is preferred.
• Experience with Databricks (Delta Lake, Spark, Unity Catalog).
• Advanced SQL and strong understanding of pyspark scripts & using pyspark for Data validation.
• Experience integrating major enterprise applications (ERP, CRM, OMS, MDM, AR systems).
• Strong understanding of data governance, data quality, metadata, and lineage.
• Excellent communication skills across business and technical audiences.

Core Technology Stack

• Analytics Platform: Databricks (Delta Lake, Unity Catalog), Microsoft PowerBI
• Languages: SQL, Spark SQL, Python / PySpark
• Governance/Data Quality: Unity Catalog/Informatica DG/DQ

About The Team

At Wesco, we build, connect, power and protect the world. As a leading provider of business-to-business distribution, logistics services and supply chain solutions, we create a world that you can depend on.

Our Company's greatest asset is our people. Wesco is committed to fostering a workplace where every individual is respected, valued, and empowered to succeed. We promote a culture that is grounded in teamwork and respect. With a workforce of over 20,000 people worldwide, we embrace the unique perspectives each person brings. Through comprehensive benefits and active community engagement, we create an environment where every team member has the opportunity to thrive.

Founded in 1922 and headquartered in Pittsburgh, Wesco is a publicly traded (NYSE: WCC) FORTUNE 500 company.

Wesco International, Inc., including its subsidiaries and affiliates ("Wesco") provides equal employment opportunities to all employees and applicants for employment. Employment decisions are made without regard to race, religion, color, national or ethnic origin, sex, sexual orientation, gender identity or expression, age, disability, or other characteristics protected by law. US applicants only, we are an Equal Opportunity Employer.

Los Angeles Unincorporated County Candidates Only: Qualified applicants with arrest or conviction records will be considered for employment in accordance with the Los Angeles County Fair Chance Ordinance and the California Fair Chance Act.

This posting is for a current, active vacancy intended for immediate hire.