Senior Data Engineer

Insight Global; Bengaluru; Hybrid; INR 3,500,000 - 6,500,000; Full time

Benefits offered by this job

Onsite three days a week

Job summary

Insight Global in Bengaluru, India, seeks a Sr. Data Engineer to design, build, and optimize modern cloud-based data platforms and analytics environments.

You will work hands-on with Snowflake, Azure Data Factory, Python, SQL, and enterprise-scale data pipelines to support critical business initiatives and AI-driven solutions.

This full-time role requires three days onsite per week.

Qualifications

• Strong hands-on Snowflake development and engineering.
• Proven experience building data pipelines with Azure Data Factory.
• ETL/ELT orchestration and production support expertise.
• Advanced ANSI SQL, stored procedures and UDFs.
• Python development for automation and Snowpark processing.
• Snowpipe and cloud-based data ingestion experience.
• Data modeling and data warehousing expertise.
• Experience with Azure and AWS services including storage and networking.
• DevOps/DataOps with CI/CD knowledge.
• Experience with PostgreSQL, MS SQL Server, or Azure SQL Database.

Responsibilities

• Design, develop, and optimize scalable Snowflake-based data solutions.
• Build and maintain high-performance ETL/ELT pipelines and cloud integrations.
• Develop automation and orchestration using Python and Snowpark.
• Implement real-time data ingestion and CDC using Snowpipe, Streams and Tasks.
• Apply data warehousing best practices and dimensional modeling.
• Manage security, governance, and compliance across data platforms.
• Collaborate with engineering and analytics teams for BI and advanced analytics.
• Support CI/CD and DataOps across the data engineering lifecycle.
• Explore AI/ML and generative AI capabilities within data ecosystems.

Skills

Snowflake

Azure Data Factory

ETL/ELT

SQL

Python

Snowpipe

Data modeling

Data warehousing

Azure/AWS

DevOps/DataOps

PostgreSQL

MS SQL Server

Azure SQL Database

Tools

Snowflake

Snowpark

Snowpipe

Azure Data Factory

Streams

Tasks

Azure SQL Database

Job description

Insight Global are seeking a Sr. Data Engineer for a leading global financial markets and technology organisation.

This position requires 3 days onsite per week in Bengaluru, India for a full time employment opportunity with Insight global.

The successful candidate will play a key role in designing, building, and optimising modern cloud-based data platforms and analytical environments. This opportunity is ideal for a hands-on data engineering professional with deep expertise in Snowflake, Azure Data Factory, Python, SQL, and enterprise-scale data pipeline development. You will support critical business initiatives, enhance platform performance, and contribute to next-generation analytics and AI-driven solutions.

Must-Have Requirements Include:

• Strong hands-on experience with Snowflake development and engineering.
• Proven experience creating and managing data pipelines using Azure Data Factory.
• Expertise in ETL/ELT orchestration and production support operations.
• Advanced SQL skills including ANSI SQL, stored procedures, and user-defined functions.
• Strong Python development experience for automation, scripting, and Snowpark-based processing.
• Experience with Snowpipe and cloud-based data ingestion frameworks.
• Data modelling and data warehousing expertise.
• Experience working with Azure and AWS, including storage and networking components.
• Strong DevOps/DataOps experience, including CI/CD automation and deployment practices.
• Experience working with relational databases such as PostgreSQL, Microsoft SQL Server, or Azure SQL Database.

Plusses Include:

• Snowflake performance tuning, including clustering keys, caching strategies, and warehouse optimisation.
• Experience managing Snowflake virtual warehouses, including scaling and multi-cluster architectures.
• Knowledge of Streams and Tasks for change data capture (CDC) and workflow scheduling.
• Experience leveraging Time Travel and Fail-Safe capabilities for data recovery and auditing.
• Strong understanding of data governance, RBAC, encryption, and secure data sharing.
• Experience integrating Snowflake with AI/ML solutions, including Snowflake Cortex and LLM-based workflows.
• Exposure to generative AI tools for accelerating data engineering and analytics initiatives.

Key Responsibilities:

• Design, develop, and optimise scalable Snowflake-based data solutions.
• Build and maintain high-performance ETL/ELT pipelines and cloud data integrations.
• Develop automation and orchestration solutions using Python and Snowpark.
• Implement real-time data ingestion and CDC frameworks using Snowpipe, Streams, and Tasks.
• Apply data warehousing best practices and dimensional modelling techniques.
• Manage security, governance, and compliance requirements across enterprise data platforms.
• Collaborate with engineering and analytics teams to support reporting, business intelligence, and advanced analytics use cases.
• Support CI/CD implementation and DataOps best practices across the data engineering lifecycle.
• Explore and implement AI/ML and generative AI capabilities within modern data ecosystems.