Senior Data Engineer

Aditi Consulting; Chennai District; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

Aditi Consulting is seeking a highly motivated Data Engineer to join our data engineering team in Chennai. The role focuses on building scalable data pipelines on AWS and Snowflake warehousing, with strong Python and SQL skills.

The ideal candidate has 7–10 years of experience, a solid background in data modeling, and hands-on work with graph and vector databases to support AI workflows. Collaboration and reliability are essential.

Qualifications

• 7-10 years of data engineering experience.
• Experience building scalable data pipelines on AWS.
• Strong Python and SQL proficiency.
• Experience with Snowflake data warehousing.
• Experience with graph and vector databases.
• Experience with Git and Agile/backlog management.

Responsibilities

• Design, develop, and maintain scalable data pipelines on AWS (S3, Glue, Lambda, EMR, Redshift).
• Build and optimize Snowflake data warehousing solutions with performance tuning.
• Write efficient Python and SQL for transformations and processing.
• Collaborate with data scientists, analysts, and stakeholders to understand data needs.
• Integrate vector databases with LLM-based AI workflows.
• Monitor, troubleshoot, and improve pipeline reliability.
• Ensure data quality, integrity, and security across pipelines.
• Participate in code reviews and architecture discussions.

Skills

AWS data pipelines

Python

SQL

Snowflake

Graph databases

Vector databases

Git

Azure DevOps

Spark / Kafka

Data modeling

Communication

Problem solving

Security/compliance

Education

Bachelor’s or Master’s in CS/Engineering

Tools

Snowflake

Neo4j

Neptune

Milvus

OpenSearch

Git

Azure DevOps

AWS Step Functions

S3

Glue

Lambda

Redshift

EMR

Kafka

Spark Streaming

CUDA RAPIDS

Job description

We are looking for a highly motivated and experienced Data Engineer to join our data engineering team. The ideal candidate will have a strong background in building scalable data pipelines using the AWS cloud stack and extensive hands-on experience with Snowflake. Proficiency in Python and SQL, along with graph and vector database technologies, is essential. This role requires strong problem-solving abilities and a proactive mindset to deliver efficient, scalable, and reliable data solutions.

Responsibilities

• Design, develop, and maintain scalable data pipelines on AWS using services such as S3, Glue, Lambda, Redshift, and EMR.
• Build and optimize data warehousing solutions using Snowflake, including performance tuning and data modeling.
• Write efficient and reusable code in Python and SQL for data transformation and processing.
• Collaborate with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data requirements.
• Integrate vector databases with LLM-based applications and AI workflows.
• Monitor, troubleshoot, and improve pipeline performance and reliability.
• Ensure data quality, integrity, and security across all stages of the pipeline.
• Participate in code reviews, architecture discussions, and continuous improvement initiatives.

Requirements

• 7-10 years of experience in data engineering or related roles.
• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.

Required Skills

• Strong hands-on experience with AWS cloud services, including data and AI workloads.
• Deep understanding of Snowflake architecture, performance tuning, and best practices.
• Advanced proficiency in Python and SQL for data pipelines, transformations, and services.
• Hands-on experience with graph databases (e.g., Neo4j, Neptune) and vector databases (e.g., Milvus, OpenSearch).
• Experience with version control systems (e.g., Git) and Git workflows.
• Experience working with Azure DevOps (AzDO) boards for backlog management in Agile environments.
• Excellent analytical and problem-solving skills.
• Strong communication and collaboration abilities.

Preferred Skills

• Experience with orchestration tools such as AWS Step Functions.
• Familiarity with data governance and compliance practices.
• Exposure to real-time data processing frameworks (e.g., Kafka, Spark Streaming).
• Knowledge of the NVIDIA ecosystem and its applications in data and AI.
• Exposure to RAPIDS libraries (cuDF, cuML, cuGraph) or CUDA-based tooling for GPU-accelerated data processing, enabling faster transformation and optimization during large-scale ingestion workflows.