Sr Data Engineer

Providence India; Hyderabad; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

You will ingest diverse data sources, optimize data architecture, monitor production pipelines, develop dashboards in Power BI, and collaborate with analytics teams in an Agile environment.

Qualifications

• 5–8 years of professional experience in Data Engineering.
• Strong programming skills in Python for ETL, data ingestion, and data transformation.
• Hands-on expertise in Azure Cloud (ADF, Azure Databricks, Azure Storage, Azure SQL, Synapse, etc.).
• Strong knowledge on Power BI and advanced analytics.
• Deep knowledge of SQL including complex queries, performance tuning, stored procedures.
• Experience with Snowflake—warehouse design and data warehousing concepts.
• Experience with structured, semi-structured (JSON, Parquet), and unstructured data.
• Experience with Agile methodologies, version control (Git), and CI/CD practices, Azure DevOps.
• Excellent analytical and problem-solving skills, with the ability to work in highly collaborative and fast paced environments.
• Exposure to AI/ML concepts and AI-powered document extraction solutions.

Responsibilities

• Design, develop, and maintain end-to-end data pipelines and ETL/ELT workflows using Python, SQL, Azure Data Factory, Snowflake, Databricks, and related services.
• Build scalable data ingestion frameworks to extract, parse, transform, and load various data types from files, APIs, databases, and applications.
• Monitor, troubleshoot, and optimize data pipelines; resolve production data issues and defects.
• Develop ad hoc and delivered reports using SQL queries and Power BI dashboards.
• Collaborate with stakeholders to understand business requirements and translate them into design documents.
• Onboard diverse data sources for integration using ADF, Databricks, Snowflake.
• Work in Agile mode and ensure code is checked in with source control.
• Stay current with data technologies and advocate improvements within the team.

Skills

Python

Azure Cloud

Power BI

SQL

Snowflake

Data Modeling

ETL/ELT

CI/CD

Git

Azure Databricks

Azure Data Factory

Azure Synapse

Azure Storage

Tools

Databricks

Snowflake

Power BI

Git

Azure DevOps

Job description

Job Description:

How is this team contributing to vision of Providence?

• Ensure a continual and seamless service for customers regardless of which services they access. What will you be responsible for?
• Design, develop, and maintain end to end data pipelines and ETL/ELT workflows using Python, SQL, Azure Data Factory, Snowflake, Databricks, and related services.
• Build scalable data ingestion frameworks to extract, parse, transform, and load structured, semi-structured, and unstructured data from files, APIs, databases, and enterprise applications.
• Drive improvements in data architecture, performance tuning, optimization, and reduce technical debt across data systems
• Monitor, troubleshoot, and enhance existing data pipelines and systems; proactively resolve data issues and production defects.
• Developing Ad hoc and delivered reports using SQL backend queries with Power BI to display the data.
• Displaying advanced analytics skills for providing insights to business owners.
• Communicate clearly when faced with unclear requirements or difficult situations so blockers can quickly be removed.
• Maintain strong documentation of data processes, ER diagrams, and architecture.
• Stay current with emerging data technologies, tools, and industry best practices, and advocate improvements within the team. What would your day look like?
• Understand business and feature requirements.
• Create design documents and architecture documents along with principal engineer and architects.
• Build and enhance data pipelines using Azure Data Factory, Databricks, Python, and Snowflake.
• Onboard different data sources for integration using ADF, Databricks, Snowflake.
• Work in Agile mode and work on assigned user stories.
• Ensure code is always checked in and ensure source control standards are followed.
• Collaborate with different teams to understand end to end requirement for new data/report.
• Working on AI use cases/initiatives. Who are we looking for?
• 5–8 years of professional experience in Data Engineering.
• Strong programming skills in Python for ETL, data ingestion, and data transformation.
• Hands on expertise in Azure Cloud (ADF, Azure Databricks, Azure Storage, Azure SQL, Synapse, etc.).
• Strong knowledge on Power BI and advanced analytics.
• Deep knowledge of SQL including complex queries, performance tuning, stored procedures.
• Experience with Snowflake—warehouse design. Solid understanding of data modeling (Star/Snowflake schemas), data warehousing concepts, workflows, and metadata management.
• Experience working with structured, semi structured (JSON, Parquet), and unstructured data.
• Strong knowledge of Agile methodologies, version control (Git), and CI/CD practices, Azure DevOps.
• Excellent analytical and problem-solving skills, with the ability to work in highly collaborative and fast paced environments.
• Experience with data testing, understanding requirements, working with product owner.
• Good to have:
• Understanding of AI/ML and Generative AI concepts, including data preparation for AI use cases.
• Exposure to Snowflake Cortex AI, Azure OpenAI, Document Intelligence, or AI-powered document extraction solutions.