Data Engineer

Cai Info India; Gurugram District; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Cai Info India is seeking a Data Engineer to design, build, and maintain our cloud data warehouse and ingestion pipelines. You will work with Snowflake, AWS DMS, and ETL/ELT processes to ingest data from Oracle, PostgreSQL, MySQL, and APIs, then model and deliver data to Spotfire and downstream applications.

The role emphasizes data quality, performance, and scalability, plus collaboration with infrastructure teams to optimize pipeline health and SLAs.

Qualifications

• Experience 3+ years building data pipelines and warehouse management (Snowflake).
• Proficient in SQL and Python for transformations and scripting.
• Experience designing and developing data ingestion from relational DBs and APIs.

Responsibilities

• Build and maintain scalable data pipelines and transformation layers within Snowflake.
• Design Snowflake objects including tables, views, stages, tasks, streams, and stored procedures.
• Apply Snowflake best practices for warehouse sizing, clustering, caching, and query optimization.
• Ensure data quality, consistency, and availability across the warehouse layers.
• Support performance tuning and scalability improvements as data volumes grow.
• Design and develop ETL/ELT pipelines to ingest data from Oracle, PostgreSQL, and MySQL databases.
• Design, develop, and maintain ETL/ELT jobs and transformations using Pentaho Data Integration (PDI/Kettle) for loading data into Snowflake.
• Configure and manage AWS DMS tasks for Full Load and CDC-based data ingestion into Snowflake.
• Build ingestion pipelines for API-based sources including JSON parsing and semi-structured data processing.
• Handle ingestion from flat files and JSON files through Snowflake external stages and COPY commands.
• Monitor pipeline health, resolve replication lag, and maintain ingestion SLAs.
• Design data models using Star Schema and Fact/Dimension structures aligned to business reporting needs.
• Use AWS S3 for staging raw data files as part of the ingestion workflow.
• Configure IAM roles and policies to manage access between AWS services and Snowflake.
• Use AWS Lambda for lightweight data transformation or pipeline trigger automation where applicable.
• Coordinate with infrastructure teams on DMS instance sizing, networking, and security group configuration.

Skills

SQL

Python

Snowflake

ETL/ELT pipelines

Data modeling (Star Schema)

API data ingestion

Tools

Pentaho Data Integration (PDI/Kettle)

AWS DMS

AWS S3

AWS Lambda

Snowflake features (stages, tasks, streams)

Job description

Role Overview

We are looking for a Data Engineer to design, build, and maintain our cloud data warehouse and ingestion pipelines. The role spans data ingestion from relational databases, APIs, and semi-structured sources via AWS DMS and custom ETL/ELT pipelines, through to data modeling and mart delivery for Spotfire and downstream business applications. The candidate will be responsible for data quality, platform performance, and scalability of the data layer.

Key Responsibilities

• Build and maintain scalable data pipelines and transformation layers within Snowflake.


• Design and develop Snowflake objects including tables, views, stages, tasks, streams, and stored procedures.


• Apply Snowflake best practices for warehouse sizing, clustering, caching, and query optimization.


• Ensure data quality, consistency, and availability across the warehouse layers.


• Support performance tuning and scalability improvements as data volumes grow.


• Design and develop ETL/ELT pipelines to ingest data from Oracle, PostgreSQL, and MySQL databases.


• Design, develop, and maintain ETL/ELT jobs and transformations using Pentaho Data Integration (PDI/Kettle) for loading data into Snowflake.


• Configure and manage AWS DMS tasks for Full Load and CDC-based data ingestion into Snowflake.


• Build ingestion pipelines for API-based sources including JSON parsing and semi-structured data processing.


• Handle ingestion from flat files and JSON files through Snowflake external stages and COPY commands.


• Monitor pipeline health, resolve replication lag, and maintain ingestion SLAs.


• Design data models using Star Schema and Fact/Dimension structures aligned to business reporting needs.


• Use AWS S3 for staging raw data files as part of the ingestion workflow.


• Configure IAM roles and policies to manage access between AWS services and Snowflake.


• Use AWS Lambda for lightweight data transformation or pipeline trigger automation where applicable.


• Coordinate with infrastructure teams on DMS instance sizing, networking, and security group configuration.

Required Skills & Experience

Core - Mandatory

• 3+ years hands-on development - pipelines, transformations, performance tuning, and warehouse management: Snowflake


• Strong proficiency in both; SQL for complex analytical queries and transformations, Python for pipeline scripting and automation: SQL and Python


• Hands-on experience designing and developing data ingestion and transformation pipelines: ETL/ELT Pipelines