Lead Data Engineer (AWS | Snowflake | Python | SQL)

Applix; Hyderabad; On-site; INR 1,400,000 - 2,100,000; Full time

Job summary

Applix in India is seeking a Senior Data Engineer to design and build scalable cloud data platforms using AWS and Snowflake. You will craft high-performance data pipelines in Python and SQL, develop data warehousing solutions, and enable AI/LLM-powered applications with graph and vector databases.

The role collaborates with data scientists and business stakeholders to deliver secure, reliable data solutions, uphold governance, and optimize pipelines.

Qualifications

• 5–8 years of professional experience in Data Engineering or related roles.
• Strong hands-on experience building scalable data solutions on the AWS Cloud Platform.
• Extensive experience designing and optimizing Snowflake data warehouse solutions.
• Advanced programming expertise in Python and SQL.
• Experience working with graph databases such as Neo4j or Amazon Neptune.
• Experience implementing vector database solutions such as Milvus or Amazon OpenSearch.
• Strong understanding of modern data architecture and distributed data processing concepts.
• Experience with version control systems and collaborative Git workflows.
• Familiarity with Azure DevOps boards for Agile project execution.
• Excellent analytical, troubleshooting, and problem-solving skills.

Responsibilities

• Design, develop, and maintain scalable, secure, and high-performance data pipelines using AWS services such as S3, Glue, Lambda, Redshift, EMR, and Step Functions.
• Develop and optimize enterprise-scale data warehousing solutions using Snowflake, including data modeling, performance tuning, and cost optimization.
• Build efficient ETL/ELT pipelines using Python and SQL for large-scale data ingestion, transformation, and processing.
• Design and implement data integration solutions across multiple structured and unstructured data sources.
• Develop and maintain graph and vector database solutions to support AI, machine learning, and LLM-based applications.
• Collaborate with data scientists, business analysts, product teams, and stakeholders to translate requirements into scalable data solutions.
• Monitor, troubleshoot, and optimize pipeline performance, reliability, and data quality.
• Implement best practices for data governance, security, compliance, and lifecycle management.
• Participate in architecture discussions, technical design reviews, and code reviews to promote engineering excellence.
• Maintain CI/CD pipelines and version-controlled repositories using Git and Agile workflows.

Skills

Python

SQL

Strong communication

Analytical thinking

Education

Bachelor's or Master's degree in Computer Science/IT/Engineering

Tools

AWS

S3

Glue

Lambda

Redshift

EMR

Step Functions

Snowflake

Neo4j

Amazon Neptune

Milvus

OpenSearch (Vector Engine)

Git

Jira

Azure DevOps (AzDO)

Docker

Kubernetes

Apache Kafka

Spark Streaming

Job description

We are seeking a highly motivated and experienced Senior Data Engineer to join our Data Engineering team. The ideal candidate will have extensive experience designing and building scalable cloud-based data platforms using the AWS ecosystem and Snowflake. This role requires expertise in developing high-performance data pipelines, data warehousing, and modern data architectures, along with strong proficiency in Python and SQL.

The successful candidate will also have experience working with graph and vector databases, enabling AI/LLM-powered applications, and will collaborate closely with data scientists, analysts, and business stakeholders to deliver reliable, scalable, and secure data solutions.

Key Responsibilities

• Design, develop, and maintain scalable, secure, and high-performance data pipelines using AWS services such as S3, Glue, Lambda, Redshift, EMR, and Step Functions.
• Develop and optimize enterprise-scale data warehousing solutions using Snowflake, including data modeling, performance tuning, and cost optimization.
• Build efficient ETL/ELT pipelines using Python and SQL for large-scale data ingestion, transformation, and processing.
• Design and implement data integration solutions across multiple structured and unstructured data sources.
• Develop and maintain graph and vector database solutions to support AI, machine learning, and LLM-based applications.
• Collaborate with data scientists, business analysts, product teams, and stakeholders to understand business requirements and translate them into scalable data solutions.
• Monitor, troubleshoot, and optimize pipeline performance, reliability, and data quality.
• Implement best practices for data governance, security, compliance, and lifecycle management.
• Participate in architecture discussions, technical design reviews, and code reviews to promote engineering excellence.
• Maintain CI/CD pipelines and version-controlled repositories using Git and Agile development practices.
• Utilize Jira or Azure DevOps (AzDO) boards for sprint planning, backlog management, and Agile project tracking.

Required Technical Skills

• Amazon Web Services (AWS)
• S3
• Glue
• Lambda
• Redshift
• EMR
• Step Functions

Data Warehousing

• Snowflake
• Data Modeling
• Performance TuningQuery Optimization

Programming

• Python
• SQL

Databases

• Graph Databases
• Neo4j
• Amazon Neptune
• Vector Databases
• Milvus
• Amazon OpenSearch (Vector Engine)
• ETL / ELT Development
• Data Transformation
• Data Integration
• Data Quality Management

DevOps & Agile

• Git
• Git Workflows

Required Qualifications

• 5–8 years of professional experience in Data Engineering or related roles.
• Strong hands-on experience building scalable data solutions on the AWS Cloud Platform.
• Extensive experience designing and optimizing Snowflake data warehouse solutions.
• Advanced programming expertise in Python and SQL.
• Experience working with graph databases such as Neo4j or Amazon Neptune.
• Experience implementing vector database solutions such as Milvus or Amazon OpenSearch.
• Strong understanding of modern data architecture and distributed data processing concepts.
• Experience with version control systems and collaborative Git workflows.
• Familiarity with Azure DevOps boards for Agile project execution.
• Excellent analytical, troubleshooting, and problem-solving skills.
• Strong verbal and written communication skills with the ability to collaborate across cross-functional teams.

Preferred Qualifications

• Experience with workflow orchestration tools such as AWS Step Functions.
• Knowledge of data governance, metadata management, and regulatory compliance practices.
• Experience with real-time data processing technologies such as Apache Kafka and Spark Streaming.
• Exposure to containerization and cloud-native deployment practices (Docker, Kubernetes) is a plus.
• Experience designing AI-ready data platforms supporting Machine Learning and Generative AI workloads.

Nice to Have

• Working knowledge of the NVIDIA AI ecosystem and GPU-accelerated computing.
• Experience with RAPIDS libraries including:
• cuDF
• cuML
• cuGraph
• Familiarity with CUDA-based technologies for accelerating large-scale data transformation and ingestion workloads.
• Exposure to LLM frameworks, Retrieval-Augmented Generation (RAG), and AI data pipelines.

Soft Skills

• Strong analytical and critical thinking abilities.
• Excellent communication and stakeholder management skills.
• Ability to work independently as well as in a collaborative team environment.
• Proactive mindset with strong ownership and accountability.
• Adaptability to work in a fast-paced, Agile development environment.
• Continuous learning attitude with a passion for emerging cloud and AI technologies.

Education

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related technical discipline.

Why Join Us?

• Work on enterprise-scale cloud data platforms and AI-driven solutions.
• Build modern data architectures leveraging AWS, Snowflake, and next-generation database technologies.
• Collaborate with cross-functional teams in a highly innovative and Agile environment.
• Opportunity to work on cutting-edge AI, machine learning, and Generative AI initiatives while driving impactful business outcomes.