Data Engineer

Aditi Consulting; Chennai District; On-site; INR 1,500,000 - 2,800,000; Full time

Job summary

Aditi Consulting is seeking a skilled Data Engineer to design and maintain scalable data pipelines on AWS, leveraging Snowflake for warehousing and integrating graph and vector databases for AI-driven use cases.

Ideal candidates will have 5–8 years of experience, strong Python/SQL skills, and experience with Neo4j/Neptune, Milvus, and OpenSearch. Collaboration with data scientists and stakeholders is essential.

Qualifications

• 5–8 years of data engineering experience.
• Strong hands-on AWS data services experience.
• Deep Snowflake architecture and tuning knowledge.
• Proficient in Python and SQL for pipelines.
• Graph and vector databases modeling experience.
• Experience with Neo4j/Neptune and Milvus/OpenSearch.
• Git and Azure DevOps for Agile backlog management.
• Bachelor’s or Master’s in CS/Engineering.

Responsibilities

• Design, develop, and maintain scalable data pipelines on AWS.
• Build and optimize data warehousing with Snowflake.
• Write Python and SQL for data transformation and processing.
• Collaborate with data scientists, analysts, and stakeholders.
• Develop and optimize graph databases (Neo4j, Neptune).
• Build vector database solutions (Milvus, OpenSearch) for AI use cases.
• Integrate vector databases with LLM-based apps and AI workflows.
• Monitor, troubleshoot, and improve pipeline performance.
• Ensure data quality, integrity, and security across pipelines.

Skills

AWS data services

Snowflake

Python

SQL

Graph databases

Vector databases

Data modeling

Data pipeline design

Git

Azure DevOps

Agile

Problem solving

Communication

Education

Bachelor’s or Master’s degree in Computer Science or Engineering

Tools

S3

Glue

Lambda

Redshift

EMR

Neo4j

Neptune

Milvus

OpenSearch

Kafka

Spark Streaming

Git

AzDO boards

Job description

We are looking for a highly motivated and experienced Data Engineer to join our data engineering team. The ideal candidate will have a strong background in building scalable data pipelines using the AWS cloud stack and extensive hands-on experience with Snowflake. Proficiency in Python and SQL, along with graph and vector database technologies, is essential. This role requires strong problem-solving abilities and a proactive mindset to deliver efficient, scalable, and reliable data solutions.

Responsibilities

• Design, develop, and maintain scalable data pipelines on AWS using services such as S3, Glue, Lambda, Redshift, and EMR.
• Build and optimize data warehousing solutions using Snowflake, including performance tuning and data modeling.
• Write efficient and reusable code in Python and SQL for data transformation and processing.
• Collaborate with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data requirements.
• Develop and optimize solutions using graph databases (e.g., Neo4j, Company Neptune), including query design and performance tuning.
• Design, build, and operate vector database solutions (e.g., Milvus, Company OpenSearch) to support semantic search, recommendations, RAG, and AI-driven use cases.
• Integrate vector databases with LLM-based applications and AI workflows.
• Monitor, troubleshoot, and improve pipeline performance and reliability.
• Ensure data quality, integrity, and security across all stages of the pipeline.
• Participate in code reviews, architecture discussions, and continuous improvement initiatives.

Requirements

• 5-8 years of experience in data engineering or related roles.
• Strong hands-on experience with AWS cloud services, including data and AI workloads.
• Deep understanding of Snowflake architecture, performance tuning, and best practices.
• Advanced proficiency in Python and SQL for data pipelines, transformations, and services.
• Strong understanding of graph and vector data modeling concepts and their practical applications.
• Hands-on experience with graph databases (e.g., Neo4j, Neptune) and vector databases (e.g., Milvus, Company OpenSearch).
• Experience with version control systems (e.g., Git) and Git workflows.
• Experience working with Azure DevOps (AzDO) boards for backlog management in Agile environments.
• Excellent analytical and problem-solving skills.
• Strong communication and collaboration abilities.
• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.

Required Skills

• Knowledge of the NVIDIA ecosystem and its applications in data and AI.
• Experience with orchestration tools such as AWS Step Functions.
• Familiarity with data governance and compliance practices.
• Exposure to real-time data processing frameworks (e.g., Kafka, Spark Streaming).