Senior Data Engineer – Integration Hub & Data Pipelines

Cuculus; Bengaluru; On-site; INR 2,500,000 - 5,000,000; Full time

Benefits offered by this job

Cutting-edge technology solutions

Innovative work environment

Job summary

A leading technology solutions provider in India is seeking a Senior Data Engineer to design, build, and manage a scalable data integration hub and data pipelines. The ideal candidate will have over 5 years of experience in data engineering, with proven skills in tools like Apache Airflow, NiFi, and Spark. You will work on end-to-end data engineering projects, ensuring data accuracy and integration across various systems. Join a dynamic team dedicated to providing innovative utility solutions.

Qualifications

• 5+ years of hands-on experience as a Data Engineer or in a similar role.
• Proven experience on at least three end-to-end data engineering projects.
• Strong hands-on experience with Apache Airflow, NiFi, Spark, Kafka and PostgreSQL.
• Proficient in Python and/or Java/Scala; Linux environment and Git.

Responsibilities

• Design, build, and maintain ETL/ELT data pipelines.
• Integrate data from multiple sources including REST APIs and files.
• Ensure high availability and fault tolerance of data workflows.
• Automate data workflows for scale and reliability.
• Develop and operate a centralized data integration hub and reusable components.
• Maintain data warehouses/lakes with proper schema evolution and governance.

Skills

Data Pipeline Design & Development

Apache Airflow

Apache NiFi

Apache Spark

Apache Kafka

PostgreSQL

Python

SQL

Debugging

Performance-tuning

Tools

Linux

Git

Spark

Kafka

PostgreSQL

Git

Job description

Overview

Shape the utilities market of the future with us! We are looking for an experienced Senior Data Engineer to design, build, and operate a scalable integration hub and data pipeline platform using modern open-source technologies. This role is hands-on and suited for an individual contributor who has delivered multiple end-to-end data engineering projects, from requirements through production deployment and operations.

You will play a critical role in enabling reliable data movement and integration across a diverse ecosystem of third-party systems, APIs, files, databases, and software platforms.

What is the role about?

Key Responsibilities

• Data Pipeline Design & Development: Design, build, and maintain robust ETL/ELT data pipelines for batch and streaming workloads.
• Implement data ingestion and transformation workflows using Apache Airflow, Apache NiFi, Apache Spark, and Kafka.
• Integrate data from multiple sources including REST APIs, files, relational databases, message queues, and external SaaS platforms.
• Optimize pipelines for performance, scalability, reliability, and cost efficiency.

• Integration Hub & Platform Engineering: Develop and operate a centralized data integration hub that supports multiple upstream and downstream systems.
• Build reusable, modular integration components and frameworks.
• Ensure high availability, fault tolerance, and observability of data workflows.

• Data Infrastructure & Storage: Design and manage data warehouses, data lakes, and operational data stores using PostgreSQL and related technologies.
• Implement appropriate data modeling strategies for analytical and operational use cases.
• Manage schema evolution, metadata, and versioning.

• Data Quality, Security & Governance: Implement data validation, monitoring, and reconciliation mechanisms to ensure data accuracy and completeness.
• Enforce data security best practices, access controls, and compliance with internal governance policies.
• Establish logging, alerting, and auditability across pipelines.

• Automation & Operations: Automate data workflows, deployments, and operational processes to support scale and reliability.
• Monitor pipelines proactively and troubleshoot production issues.
• Improve CI/CD practices for data engineering workflows.

• Collaboration & Stakeholder Engagement: Work closely with data scientists, analysts, backend engineers, and business stakeholders to understand data requirements.
• Translate business needs into technical data solutions.
• Provide technical guidance and best practices across teams.

Required Skills And Qualifications

• 5+ years of hands-on experience as a Data Engineer or in a similar role.
• Proven experience as an individual contributor on at least three end-to-end data engineering projects, from design to production.
• Strong hands-on experience with:
• Apache Airflow / Dagster
• Apache NiFi
• Apache Spark
• Apache Kafka
• PostgreSQL
• Extensive experience integrating data from APIs, files, databases, and third-party systems.
• Strong SQL skills and experience with data modeling.
• Solid programming experience in Python and/or Java/Scala.
• Experience with Linux environments and version control systems (Git).
• Strong problem-solving, debugging, and performance-tuning skills.

• Preferred Skills And Qualifications

• Experience with cloud platforms (AWS, Azure, or GCP).
• Experience with containerization and orchestration (Docker, Kubernetes).
• Knowledge of data lake technologies and formats (Parquet, ORC, Iceberg, Delta Lake).
• Familiarity with monitoring and observability tools for data pipelines.
• Experience working in fast-paced or startup environments.

About Us

Cuculus is the key to providing utilities to all, while protecting the world’s precious resources. Jointly with our international partner network, we provide cutting-edge software and technology solutions to address utility challenges now and in the future. Cuculus will never tire of creating innovative technology and services that enable utilities and organisations to successfully transition to a new era of providing and managing electricity, water, and gas. The work we do is important for individuals, cities, and entire nations. Our work is serious, but we have fun with it, too.