Data Engineer

Affine Analytics; Bengaluru; Hybrid; INR 2,500,000 - 4,200,000; Full time

Job summary

Affine Analytics is seeking an experienced Data Engineer to design and optimize ETL/ELT pipelines for batch and streaming data in a cloud-first environment. You will build and maintain data warehouses and lakes using Snowflake, BigQuery and Redshift, collaborating with data scientists and product teams to deliver reliable data assets.

You will ensure data quality, observability, and performance, while shaping data models and overseeing CI/CD driven workflows with tools like Airflow, Dagster and

Qualifications

• 5+ years of experience in data engineering or a related role.
• Proficiency in SQL and Python (or Scala/Java).
• Experience with cloud platforms (AWS).
• Hands-on with big data tools: Spark, Kafka, dbt, or similar.
• Familiarity with data warehousing concepts and dimensional modeling.
• Strong understanding of CI/CD and version control (Git). Nice to Have
• Experience with real-time/streaming pipelines (Kafka, Flink, Kinesis).
• Exposure to ML pipelines or MLOps tooling.
• Knowledge of data governance and data catalog tools (e.g., Atlan, Alation).

Responsibilities

• Design, build, and optimize ETL/ELT pipelines for batch and streaming data.
• Develop and maintain data warehouses and data lakes (Snowflake, BigQuery, Redshift).
• Collaborate with data scientists, analysts, and product teams to understand data needs.
• Ensure data quality, reliability, and observability across pipelines.
• Build and manage orchestration workflows (Airflow, Prefect, Dagster).
• Implement best practices for data modeling, partitioning, and performance tuning.
• Monitor and troubleshoot data infrastructure issues in production.

Skills

5+ years experience

SQL proficiency

Python/Scala/Java

AWS

CI/CD & Git

Data warehousing concepts

Real-time streaming

ML pipelines / MLOps

Data governance & catalog tools

Tools

Spark

Kafka

dbt

Airflow

Dagster

Job description

Responsibilities

• Design, build, and optimize ETL/ELT pipelines for batch and streaming data
• Develop and maintain data warehouses and data lakes (e.g., Snowflake, BigQuery, Redshift)
• Collaborate with data scientists, analysts, and product teams to understand data needs
• Ensure data quality, reliability, and observability across pipelines
• Build and manage orchestration workflows (e.g., Airflow, Prefect, Dagster)
• Implement best practices for data modeling, partitioning, and performance tuning
• Monitor and troubleshoot data infrastructure issues in production

Key Requirements

• 5+ years of experience in data engineering or a related role
• Proficiency in SQL and Python (or Scala/Java)
• Experience with cloud platforms (AWS)
• Hands-on with big data tools: Spark, Kafka, dbt, or similar
• Familiarity with data warehousing concepts and dimensional modeling
• Strong understanding of CI/CD and version control (Git) Nice to Have
• Experience with real-time/streaming pipelines (Kafka, Flink, Kinesis)
• Exposure to ML pipelines or MLOps tooling
• Knowledge of data governance and data catalog tools (e.g., Atlan, Alation)