Data Engineer II Mid

D8 LooP; Bengaluru; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

D8 LooP in Bengaluru is seeking a data engineer to architect and build scalable data pipelines that ingest data from diverse sources, ensure governance, and support AI/ML insights. You will collaborate with a team of ML scientists and engineers, owning end-to-end data flows, optimizing performance and cost, and delivering reliable data contracts for downstream users.

The role demands strong SQL, warehousing, and AWS data services experience, plus hands-on experience with real-time data

Qualifications

• Experience with data modeling, warehousing and building ETL pipelines.
• Proficiency with SQL and data warehousing architectures.
• Hands-on AWS data services such as Redshift, S3, Glue, EMR, Kinesis and Lambda.
• Knowledge of data governance, security, and cost/performance optimization.
• Experience in real-time data processing for AI/ML workflows is a plus.
• Ability to design scalable data systems with data contracts and APIs.
• Experience with Airflow for orchestration.
• Familiarity with data quality, metadata management and entity resolution.

Responsibilities

• Design scalable data systems compliant with privacy and security.
• Build and maintain ETL pipelines and data models.
• Collaborate with ML and BI teams to enable AI-driven insights.
• Define data contracts and API standards for downstream consumers.
• Optimize AWS infrastructure for performance and cost.

Skills

Data modeling

Data warehousing

ETL pipelines

SQL

AWS data services

Real-time data processing

Data governance

Data quality

API design & contracts

Multi-system integration

Spark

Tools

Redshift

S3

AWS Glue

EMR

Kinesis

Firehose

Lambda

IAM roles and permissions

Airflow

Spark

Job description

Are you passionate about working with data at scale? Do you love bringing data together from diverse systems and sources and working on critical problems for understanding customer behaviour and generating actionable insights? Does the idea of partnering with a team of highly experienced machine-learning scientists and engineers excite you?

Skills, with evidence

Experience with data modeling, warehousing and building ETL pipelines must have

• SQL 4+ years of SQL experience must have
  • Warehousing Experience with data modeling, warehousing and building ETL pipelines must have
    • AWS Experience with AWS technologies like Redshift, S3, AWS Glue, EMR, Kinesis, FireHose, Lambda, and IAM roles and permissions must have · not practised here
      • Governance & security As a Data Engineer on the Minerva Athena team, you will own and drive the design of scalable data systems keeping privacy and security compliance in check. not practised here
        • Cost & performance Architect and optimise AWS infrastructure: EC2, Lambda, S3, Redshift, EMR- balancing performance, reliability, and cost
          • Streaming Design and architect AI-native infrastructure supporting real-time data processing for AI/ML inference, training, and continuous learning at scale.
            • Airflow / orchestration Architect infrastructure for agentic AI systems with multi-agent orchestration, defining patterns and best practices for the team.
              • Drive GenAI-powered data quality, entity resolution, and metadata management strategies that raise the bar for data integrity
              • Design high-throughput, fault-tolerant pipelines supporting BI Engineers, data/applied scientists, and AI agents at scale.
              • Define data contracts and API standards for reliable, versioned data consumption across downstream consumers
              • Spark - Experience with AWS technologies like Redshift, S3, AWS Glue, EMR, Kinesis, FireHose, Lambda, and IAM roles and permissions
              • Machine learning Minerva is a cross-functional team of highly experienced Applied scientists, Data Scientists, Data Engineers, BI Engineers and Software Engineers with a critical business mission making revolutionary leaps forward using massive-scale data with advanced analytics and machine learning and helping democratise the publishing industry. not practised here