Data Engineer

Halcer; Bengaluru; On-site; INR 2,500,000 - 4,200,000; Full time

Job summary

Halcer is seeking a Data Engineer with 6+ years of experience to design and maintain scalable data pipelines for real-time and batch processing. You will work with Kafka, Spark, Airflow, and cloud platforms to support analytics, forecasting, and ML workloads.

You will partner with AI/ML teams, DevOps, and data scientists to enable clean, ML-ready datasets, while deploying data infrastructure across on-prem and hybrid environments.

Qualifications

• 6+ years in core data engineering roles.
• 2+ years building real-time/streaming pipelines.
• Bachelor’s or Master’s in Computer Science, Engineering, or related field.

Responsibilities

• Design real-time (Kafka, Spark/Flink) and batch pipelines for high-throughput ingestion and transformation.
• Develop data models and manage data lakes/warehouses (Delta Lake, Iceberg, etc.).
• Integrate data from IoT sensors, databases, REST APIs, and flat files.
• Ensure pipelines are scalable, observable, and high quality with lineage tracking.
• Collaborate with AI/ML teams to provision ML-ready datasets.
• Deploy, optimize, and manage pipelines across on-prem and hybrid environments.
• Participate in architecture decisions for resilient, cost-effective data flows.
• Contribute to IaC for data deployment (Terraform, Ansible).

Skills

Python

Java

SQL

Apache Kafka

Apache Spark

Apache Flink

Airflow

dbt

Parquet

Avro

ORC

AWS

GCP

Azure

Snowflake

BigQuery

Redshift

Delta Lake

Docker

Kubernetes

Terraform

Ansible

Great Expectations

OpenMetadata

InfluxDB

TimescaleDB

Education

Bachelor's or Master's in CS/Engineering

Tools

Airflow

dbt

Terraform

Ansible

Docker

Kubernetes

OpenMetadata

Job description

About Halcer

Halcer is a leading IT services, consulting, and staffing organization committed to delivering innovative technology solutions and top-tier technical talent to global enterprises.

Job Overview

We are seeking an experienced Data Engineer (6+ Years of Experience) to build and maintain scalable, high-performance data pipelines and infrastructure for our next-generation data platform. The platform ingests and processes real-time and historical data from diverse industrial sources such as airport systems, sensors, cameras, and APIs. You will work closely with AI/ML engineers, data scientists, and DevOps teams to enable reliable analytics, forecasting, and anomaly detection use cases.

Key Responsibilities

• Design and implement real-time (Kafka, Spark/Flink) and batch (Airflow, Spark) pipelines for high-throughput data ingestion, processing, and transformation.
• Develop data models and manage data lakes and warehouses (Delta Lake, Iceberg, etc.) to support both analytical and ML workloads.
• Integrate data from diverse sources: IoT sensors, databases (SQL/NoSQL), REST APIs, and flat files.
• Ensure pipeline scalability, observability, and data quality through monitoring, alerting, validation, and lineage tracking.
• Collaborate with AI/ML teams to provision clean and ML-ready datasets for training and inference.
• Deploy, optimize, and manage pipelines and data infrastructure across on-premise and hybrid environments.
• Participate in architectural decisions to ensure resilient, cost-effective, and secure data flows.
• Contribute to infrastructure-as-code and automation for data deployment using Terraform, Ansible, or similar tools.

Qualifications & Experience Requirements

• Total Experience: 6+ years of total experience in core data engineering roles.
• Streaming & Real-Time Experience: 2+ years of direct experience building and maintaining real-time or streaming pipelines.
• Education: Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.

Required Skills

• Strong programming proficiency in Python or Java, alongside expert-level SQL.
• Hands-on experience with Apache Kafka, Apache Spark, or Apache Flink for real-time and batch processing.
• Proficiency with workflow orchestration tools like Airflow, dbt, or similar technologies.
• Deep familiarity with data modeling (OLAP/OLTP), schema evolution, and file formats (Parquet, Avro, ORC).
• Hands-on experience with hybrid/on-premise and cloud platform deployments (AWS, GCP, or Azure).
• Proven track record working with data lakes and modern data warehouses (Snowflake, BigQuery, Redshift, or Delta Lake).
• Strong working knowledge of DevOps practices, Docker, Kubernetes, and IaC tools like Terraform or Ansible.
• Knowledge of data observability, data cataloging, and quality frameworks (e.g., Great Expectations, OpenMetadata).

Good-to-Have Skills

• Experience with time-series databases (e.g., InfluxDB, TimescaleDB) and processing high-frequency sensor data.
• Prior domain experience in aviation, manufacturing, or logistics.