Data Engineer

Insight Global; Hyderabad; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

Insight Global is seeking a Data Engineer to support the Market Basket custom application and build reliable data ingestion, transformation, and orchestration pipelines using Azure Data Factory, Databricks, SQL Server / Azure SQL, and BI foundations.

You will collaborate with application, backend, product, QA, DevOps, and business stakeholders to deliver governed, trusted data assets, create reporting-ready datasets, and ensure data quality with monitoring and logging across the data platform.

Job description

Job Description

We are seeking a Data Engineer to support the Market Basket custom application and establish the foundational data

capabilities needed for analytics, reporting, and operational visibility. This role will focus on building reliable data

ingestion, transformation, and orchestration pipelines using Azure Data Factory, Databricks, SQL Server / Azure SQL,

and BI reporting foundations. The engineer will partner with application, backend, product, QA, DevOps, and business

stakeholders to deliver governed, trusted, and reusable data assets.

• Design, build, and maintain data pipelines using Azure Data Factory and related Azure data services.
• Develop data ingestion, transformation, validation, and orchestration workflows across source systems and reporting stores.
• Build scalable data processing logic using Databricks notebooks, jobs, and Spark-based transformation patterns.
• Design and optimize SQL Server / Azure SQL tables, views, stored procedures, data marts, and reporting-ready datasets.
• Create foundational data models and curated datasets to support BI reporting, dashboards, and operational insights.
• Implement data quality checks, reconciliation routines, exception handling, logging, and pipeline monitoring.
• Collaborate with backend and application teams to align application events, transactional data, and reporting needs.
• Partner with business stakeholders to understand reporting requirements, KPI definitions, data availability, and refresh expectations.
• Support CI/CD, environment configuration, deployment validation, and production troubleshooting for data pipelines.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to [e-mail] learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.

Skills and Requirements

• Strong hands-on experience with Azure Data Factory pipeline development, orchestration, triggers, linked services, and integration runtimes.
• Experience building data pipelines, ELT/ETL workflows, data validation routines, and operational pipeline monitoring.
• Hands-on experience with Databricks, Spark, notebooks, jobs, and data transformation patterns.
• Strong SQL skills with SQL Server / Azure SQL, including data modeling, stored procedures, views, query tuning, and data quality validation.
• Experience designing curated datasets, data marts, or semantic/reporting layers for BI reporting use cases.
• Understanding of data governance, lineage, security, access controls, and environment-specific configuration practices.
• Experience working with Azure DevOps, Git, CI/CD workflows, and release support for data engineering assets.
• Ability to collaborate with application, backend, reporting, QA, DevOps, and business teams in an Agile delivery model.
• Strong analytical, troubleshooting, documentation, and communication skills.
• Experience with Microsoft Power BI dataset enablement, reporting models, and dashboard support.
• Exposure to Azure Data Lake Storage, Delta Lake, medallion architecture, or lakehouse implementation patterns.
• Experience with incremental loads, CDC-style patterns, batch processing, and data reconciliation frameworks.
• Familiarity with observability and monitoring using Azure Monitor, Log Analytics, or pipeline alerting approaches.
• Experience supporting production data platforms and reporting foundations in enterprise environments.