Data Engineer / AI Engineer

Aventra.AI; Bengaluru; Hybrid; INR 4,000,000 - 6,000,000; Full time

Job summary

Aventra.AI is seeking a Senior Data Engineer / AI Engineer in Bengaluru to design and implement scalable data pipelines and AI-powered data solutions. You will handle structured, semi-structured, and unstructured data across cloud platforms, leveraging Spark, PySpark, and AI frameworks.

You will build ETL/ELT workflows, real-time streaming, and data orchestration, collaborating with data scientists and architects to ensure robust data quality, governance, and security in enterprise environments.

Qualifications

• 8+ years of experience in Data Engineering, Big Data, or AI Engineering.
• Strong hands-on Python, PySpark, and SQL.
• Extensive Big Data tech exp: Spark, Hadoop, Kafka, Hive.
• Understanding of distributed computing and large-scale processing.
• Experience building and optimizing complex ETL/ELT pipelines.
• Knowledge of AI/ML concepts and integration with data platforms.
• Experience with cloud-based data platforms.

Responsibilities

• Design, develop, and maintain scalable data pipelines.
• Build robust ETL/ELT workflows for structured and unstructured data.
• Work with large-scale distributed data processing frameworks.
• Develop ingestion, transformation, validation, and orchestration workflows.
• Optimize data pipelines for performance, scalability, reliability, and cost.
• Implement batch and real-time data processing solutions.
• Work with Spark, PySpark, Hadoop, Hive, and Kafka.
• Develop distributed data processing apps using Spark/PySpark.
• Implement real-time streaming with Kafka or similar tech.
• Work with data lakes, data warehouses, and lakehouse architectures.
• Optimize SQL queries, data models, and large-scale processing jobs.
• Experience with Delta Lake, Iceberg, or Hudi is a plus.

Skills

Python

PySpark

SQL

Apache Spark

Hadoop

Hive

Kafka

ETL/ELT

Data Pipelines

Data Warehousing

AI Frameworks

Vector Databases

Cloud

Orchestration

Databases

Tools

Delta Lake

Apache Iceberg

Hudi

AWS S3

Azure Data Factory

Google BigQuery

Job description

Experience: 8+ Years

Work Mode: Hybrid (Work From Office)

Role Type: Full-Time

Job Summary

We are looking for an experienced Senior Data Engineer / AI Engineer with strong expertise in Big Data technologies, data engineering, distributed computing, and AI/ML engineering.

The ideal candidate should have hands‑on experience designing and developing scalable data pipelines, processing large volumes of structured and unstructured data, and building AI‑powered data solutions. The candidate should possess strong programming skills, experience with modern Big Data ecosystems, and the ability to integrate AI/ML capabilities into enterprise data platforms.

• Design, develop, and maintain scalable and high-performance data pipelines.
• Build robust ETL/ELT workflows for processing structured, semi‑structured, and unstructured data.
• Work with large-scale distributed data processing frameworks.
• Develop data ingestion, transformation, validation, and orchestration workflows.
• Optimize data pipelines for performance, scalability, reliability, and cost efficiency.
• Implement batch and real‑time data processing solutions.
• Work extensively with Apache Spark, PySpark, Hadoop, Hive, and Kafka.
• Develop distributed data processing applications using Spark and PySpark.
• Implement real‑time streaming solutions using Kafka or similar technologies.
• Work with data lakes, data warehouses, and lakehouse architectures.
• Optimize SQL queries, data models, and large‑scale data processing jobs.
• Experience with Delta Lake, Apache Iceberg, or Hudi is an added advantage.

3. AI / Machine Learning Engineering

• Design and develop AI/ML solutions integrated with enterprise data platforms.
• Build and deploy machine learning models and data‑driven applications.
• Work with Python‑based AI/ML frameworks and libraries.
• Develop AI‑powered workflows using LLMs, Generative AI, and related technologies.
• Integrate LLM APIs and AI capabilities into existing data pipelines and enterprise applications.
• Work on RAG‑based solutions, embeddings, vector databases, and semantic search.
• Implement model evaluation, monitoring, and performance optimization.
• Hands‑on experience with cloud‑based data platforms such as AWS, Microsoft Azure, or Google Cloud Platform (GCP).
• Experience with services such as AWS S3, Glue, EMR, Redshift, Azure Data Factory, Azure Databricks, or Google BigQuery.
• Build and manage scalable cloud‑native data engineering solutions.
• Implement data storage, processing, and analytics solutions across cloud environments.
• Design scalable data architectures, data models, and data integration frameworks.
• Implement data quality checks, data validation, and data governance practices.
• Ensure data security, privacy, and compliance across data platforms.
• Work closely with data scientists, AI engineers, architects, and business stakeholders.

Required Technical Skills

Skill CategoryRequired Skills

Programming

Python, PySpark, SQL

Apache Spark, Hadoop, Hive

Streaming

ETL/ELT, Data Pipelines, Data Warehousing

AI/ML

AI Frameworks

Vector Databases

Cloud

Orchestration

Databases

Mandatory Requirements

• 8+ years of overall experience in Data Engineering, Big Data Engineering, or AI Engineering.
• Strong hands‑on experience with Python, PySpark, and SQL.
• Extensive experience working with Big Data technologies such as Spark, Hadoop, Kafka, and Hive.
• Strong understanding of distributed computing and large‑scale data processing.
• Experience developing and optimizing complex ETL/ELT data pipelines.
• Good understanding of AI/ML concepts and their integration with data engineering solutions.
• Experience working with cloud‑based data platforms.
• Strong analytical, problem‑solving, and communication skills.