Data Engineer

Pi Square Technologies; India; Hybrid; INR 1,200,000 - 2,100,000; Full time

Job summary

Pi Square Technologies is seeking a Spark-focused data engineer in India to design, build, and maintain scalable data pipelines. You will work with Spark Core, Structured Streaming, Spark SQL, DataFrames, and the Java Spark API to deliver batch and real-time analytics solutions.

You will optimize performance, handle large datasets, and ensure data quality while collaborating with architects, engineers, and analysts to meet business needs.

Qualifications

• Hands-on experience with Apache Spark across Core, SQL and Structured Streaming.
• Experience building scalable ETL/ELT data pipelines in Spark.
• Strong Java experience with Spark API and multithreading.

Responsibilities

• Design, develop, and maintain scalable data processing pipelines using Apache Spark.
• Develop batch and real-time data processing solutions using Spark Core and Structured Streaming.
• Work with Spark SQL, DataFrames, and Dataset APIs.
• Develop Spark applications using the Java Spark API.
• Build reliable ETL/ELT data pipelines and optimize performance.
• Tune Spark performance for large-scale workloads and troubleshoot issues.
• Implement concurrency and multithreading for efficient processing.
• Use Java Streams API for data processing and transformation.
• Design data models for analytical and operational use cases.
• Collaborate with data architects, software engineers, and analysts.
• Monitor production pipelines and address data-quality issues.

Skills

Apache Spark

Spark Core

Spark SQL

Spark Structured Streaming

Spark DataFrames

Spark Dataset API

Java Spark API

Spark Performance Tuning

Concurrency

Java Streams API

Job description

Role & responsibilities

• Design, develop, and maintain scalable data processing pipelines using Apache Spark.
• Develop batch and real-time data processing solutions using Spark Core and Structured Streaming.
• Work extensively with Spark SQL, DataFrames, and Dataset APIs.
• Develop Spark applications using the Java Spark API.
• Build reliable and high-performance ETL/ELT data pipelines.
• Perform Spark performance tuning and optimization for large-scale workloads.
• Troubleshoot Spark jobs related to performance, memory, serialization, partitioning, and resource utilization.
• Implement efficient solutions using concurrency and multithreading.
• Use Java Streams API for efficient data processing and transformation.
• Design and implement appropriate data models for analytical and operational use cases.
• Work with large datasets and distributed processing environments.
• Ensure data quality, reliability, scalability, and performance of data pipelines.
• Collaborate with data architects, software engineers, analysts, and other stakeholders.
• Monitor and troubleshoot production data pipelines and resolve performance or data-quality issues.
• Follow engineering best practices around code quality, testing, deployment, monitoring, and documentation.

Mandatory Skills

Apache Spark

Strong hands-on experience with:

• Apache Spark Core
• Spark SQL
• Spark Structured Streaming
• Spark DataFrames
• Spark Dataset API
• Java Spark API
• Spark Performance Tuning
• Concurrency & Multithreading
• Java Streams API