Senior Data Engineer

Insight Global; Bengaluru; On-site; INR 3,500,000 - 6,000,000; Full time

Job summary

Insight Global is seeking an experienced Snowflake Data Engineer to design, build and optimise modern data pipelines and cloud-based analytics environments in Bengaluru. The role focuses on Snowflake features, SQL, Python, and cloud platforms like Azure to support enterprise data initiatives and enable advanced analytics including AI/ML integrations.

You will architect data models, manage Snowflake virtual warehouses, implement Streams, Tasks, Time Travel, and develop robust ETL/ELT pipelines

Qualifications

• 8+ years of experience in data engineering.
• Strong hands‑on experience in ETL/ELT orchestration and production operations.
• Hands on Snowflake experience.
• Proficiency with SQL: Advanced ANSI SQL, stored procedures, functions.
• Proficiency with Python: Scripting, automation, Snowpark usage.
• Experience building data pipelines using Snowpipe.
• Expertise with Azure, AWS (storage, networking).
• Experience with OLTP databases such as PostgresDB or MS SQL Server or Azure SQL.

Responsibilities

• Design, build and optimise high‑performance Snowflake pipelines.
• Manage Snowflake virtual warehouses with sizing, scaling and multi‑cluster configs.
• Tune workloads via clustering keys, caching, and query tuning.
• Implement Streams and Tasks for real‑time ingestion and CDC.
• Utilise Time Travel and Fail‑Safe for data recovery and auditing.
• Develop ETL/ELT pipelines including Snowpipe ingestion.
• Use Python for scripting and in‑database processing via Snowpark.
• Integrate Snowflake with Azure and AWS storage, networking and compute resources.

Skills

SQL

Python

Snowflake

Azure

AWS

Data pipelines

CI/CD

Tools

Snowpipe

Snowpark

Job description

Insight Global is looking for an experienced Snowflake Data Engineer to design, build, and optimise modern data pipelines and cloud-based analytical environments. The ideal candidate will have strong expertise in SQL, Python, Snowflake features (Warehouses, Streams, Tasks, Time Travel), and cloud platforms such as Azure. This role will support enterprise data initiatives, ensure high performance, and enable advanced analytics including AI/ML integrations.

Key Responsibilities

Snowflake Development & Engineering

• Develop high‑performance SQL including advanced ANSI SQL, stored procedures, and user-defined functions.
• Build and manage Snowflake virtual warehouses, ensuring correct sizing, scaling, and multi‑cluster configurations.
• Optimise Snowflake workloads using clustering keys, caching strategies, warehouse configuration, and query tuning.
• Implement Streams and Tasks for real‑time ingestion, change data capture (CDC), and automated scheduling.
• Leverage Time Travel and Fail‑Safe features for data recovery, auditing, and historical analysis.
• Build and maintain ETL/ELT pipelines, including Snowflake ingestion using Snowpipe.
• Use Python for scripting, workflow automation, and in‑database processing through Snowpark.
• Integrate Snowflake with cloud storage, networking, and compute resources across Azure and AWS.

Data Architecture & Warehousing

• Apply strong data warehousing principles and dimensional modelling techniques.
• Ensure data models support analytics, reporting, and scalable business intelligence use cases.

Security, Governance & Compliance

• Implement data governance practices including RBAC, data access controls, encryption, and secure data sharing.

AI/ML & Emerging Technologies

• Integrate Snowflake with AI/ML solutions, including Snowflake Cortex and large language model (LLM) workflows.
• Use generative AI tools to automate data engineering and accelerate development tasks.

DevOps/DataOps

• Build and maintain CI/CD pipelines supporting data engineering workflows.
• Automate deployment, testing, and environment management using modern DevOps/DataOps practices.

Must Haves:

8 + years of experience in data engineering

Strong hands‑on experience in ETL/ELT orchestration and production operations

Hands on Snowflake experience

Proficiency with SQL: Advanced ANSI SQL, stored procedures, functions

Proficiency with Python: For scripting, automation, and using Snowpark for in‑database processing

Experience building data pipelines using tools like Snowpipe

Expertise with Azure, AWS (storage, networking)

Experience with OLTP databases such as PostgresDB or MS SQL Server or Azure SQL

Pluses:

Performance Tuning: Caching, clustering keys, virtual warehouses

Data Governance: Role-Based Access Control (RBAC), data sharing, encryption

Virtual Warehouses: Sizing, scaling, multi-cluster warehouses

Streams & Tasks: Change data capture and scheduling

Time Travel & Fail-Safe: Data recovery and historical access

AI/ML Integration: Using Snowflake Cortex, integrating LLMs

Generative AI Tools: Leveraging AI for data tasks