Data Engineer

WillWare Technologies; Chennai District; On-site; INR 1,200,000 - 2,500,000; Full time

Benefits offered by this job

AI-driven ML pipelines

Job summary

WillWare Technologies is seeking a Data Engineer with 5+ years of hands-on experience to design and maintain scalable data pipelines across batch and streaming workloads. The role emphasizes Python, SQL, Airflow, and cloud platforms.

Based in India with onsite work mode, the position involves collaborating with analytics and platform engineering teams to deliver reliable data solutions using Spark, Beam, Flink, dbt, and GCP.

Qualifications

• Strong Python and SQL skills required.
• Experience with Apache Airflow for workflow orchestration.
• Hands-on experience with Spark, Beam or Flink.
• Proficiency with dbt and data transformations.
• Cloud experience, preferably GCP.
• Streaming data experience with Kafka or Pub/Sub.
• Familiarity with AI-assisted coding tools.

Responsibilities

• Build and maintain scalable batch and streaming data pipelines.
• Develop distributed data processing using Spark, Beam, or Flink.
• Orchestrate workflows using Apache Airflow.
• Develop data transformations using Python, SQL, and dbt.
• Work with GCP or other major cloud platforms.
• Ensure data quality, performance, and pipeline reliability.
• Collaborate with Analytics and Platform Engineering teams.

Skills

Python & SQL

Apache Airflow

Spark / Beam / Flink

dbt

GCP / Cloud

Kafka / Pub/Sub

AI-assisted coding tools

Tools

Apache Airflow

Spark / Beam / Flink

dbt

GCP

Kafka / Pub/Sub

GitHub Copilot/Cursor

Gemini Code Assist

Job description

Role: Data Engineer

Experience: 5+ Years

Role: Data Engineer

Location:Bangalore, Chennai, Hyderabad, Pune, Kolkata

Work Mode: Onsite

Key Responsibilities:

• Build and maintain scalable batch and streaming data pipelines.
• Develop distributed data processing using Spark, Apache Beam, or Flink.
• Orchestrate workflows using Apache Airflow.
• Develop data transformations using Python, SQL, and dbt.
• Work with GCP or other major cloud platforms.
• Ensure data quality, performance, and pipeline reliability.
• Collaborate with Analytics and Platform Engineering teams.

Mandatory Skills:

• Strong Python & SQL
• Apache Airflow
• Spark / Apache Beam / Flink
• dbt
• Cloud experience, preferably GCP
• Streaming: Kafka / Pub/Sub
• Hands-on experience with AI-assisted coding tools such as GitHub Copilot, Cursor, Gemini Code Assist, etc.

Good to Have:

AI-driven data/ML pipelines, data quality/governance, BI/analytics experience.