Data Engineer

NuStar Technologies Pte Ltd; Pune District, Chennai, Bengaluru; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

NuStar Technologies Pte Ltd is seeking an experienced Data Engineer to design, build, and maintain scalable batch and streaming data pipelines. You will work with Spark, Flink, Kafka, and SQL to shape data models and deliver reliable data access across distributed systems.

The role emphasizes collaboration with data scientists and business teams to ensure data quality, governance, and scalable analytics. Experience with cloud platforms and data lakehouse concepts is a plus.

Qualifications

• Strong hands-on experience with Apache Spark and PySpark.
• Experience with Apache Flink for real-time stream processing.
• Good knowledge of Apache Kafka, including topics, partitions, producers, consumers, and event streaming.
• Strong SQL skills and experience working with large datasets.
• Experience in data modeling, including conceptual, logical, and physical data models.
• Knowledge of ETL/ELT processes and data pipeline development.
• Understanding of distributed computing, parallel processing, and data processing optimization.
• Experience with data quality, data validation, and troubleshooting pipeline issues.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Apache Spark, PySpark, and Apache Flink.
• Build real-time data ingestion and streaming solutions using Apache Kafka.
• Develop batch and stream-processing applications for large-scale data processing.
• Design, implement, and maintain data models to support analytical and business reporting requirements.
• Perform data transformation, cleansing, aggregation, and validation across multiple data sources.
• Optimize Spark jobs, streaming applications, SQL queries, and data processing workflows for performance.
• Work with structured and semi-structured data in distributed computing environments.
• Implement data quality checks, error handling, monitoring, and pipeline reliability mechanisms.
• Collaborate with data architects, data analysts, data scientists, and business teams to understand data requirements.
• Follow best practices for data governance, security, scalability, and documentation.

Skills

Real-time streaming

Batch processing

Data modeling

SQL querying

Problem solving

Tools

Apache Spark

PySpark

Apache Flink

Apache Kafka

SQL

Job description

Job Summary

We are looking for an experienced Data Engineer with strong expertise in distributed data processing, real-time streaming, and data modeling. The ideal candidate should have hands-on experience with Apache Spark, Apache Flink, Kafka, and SQL to design, develop, and maintain scalable batch and streaming data pipelines.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Apache Spark, PySpark, and Apache Flink.
• Build real-time data ingestion and streaming solutions using Apache Kafka.
• Develop batch and stream-processing applications for large-scale data processing.
• Design, implement, and maintain data models to support analytical and business reporting requirements.
• Perform data transformation, cleansing, aggregation, and validation across multiple data sources.
• Optimize Spark jobs, streaming applications, SQL queries, and data processing workflows for performance.
• Work with structured and semi-structured data in distributed computing environments.
• Implement data quality checks, error handling, monitoring, and pipeline reliability mechanisms.
• Collaborate with data architects, data analysts, data scientists, and business teams to understand data requirements.
• Follow best practices for data governance, security, scalability, and documentation.

Required Skills

• Strong hands-on experience with Apache Spark and PySpark.
• Experience with Apache Flink for real-time stream processing.
• Good knowledge of Apache Kafka, including topics, partitions, producers, consumers, and event streaming.
• Strong SQL skills and experience working with large datasets.
• Experience in data modeling, including conceptual, logical, and physical data models.
• Knowledge of ETL/ELT processes and data pipeline development.
• Understanding of distributed computing, parallel processing, and data processing optimization.
• Experience with data quality, data validation, and troubleshooting pipeline issues.

Good to Have

• Experience with cloud data platforms such as AWS, Azure, or GCP.
• Knowledge of data lakes, data warehouses, and lakehouse architectures.
• Familiarity with Delta Lake, Apache Iceberg, or Apache Hudi.
• Experience with workflow orchestration tools such as Airflow.
• Knowledge of schema management and formats such as Avro, Parquet, and JSON.
• Experience with data governance, metadata management, and data cataloguing.

Candidate Profile

The ideal candidate should have practical experience building scalable batch and real-time data processing solutions using Spark, Flink, and Kafka, along with strong data modeling and SQL capabilities.

Key Skills for Sourcing: Data Engineer, Apache Spark, PySpark, Apache Flink, Kafka, Data Modeling, SQL, ETL/ELT, Real-Time Streaming, Batch Processing.