Java Spark Data Engineer

Wissen Technology; Pune District; On-site; INR 1,200,000 - 2,100,000; Full time

Job summary

Wissen Technology seeks a data engineer to build and maintain data transformation pipelines using Java Spark in Pune. You will optimize large-scale data processing, orchestrate workflows with Airflow, and implement data quality checks with a focus on metadata and lineage.

The role requires 3+ years of experience in data engineering, strong Spark and HBase knowledge, and familiarity with cloud or container platforms. Pune-based with potential hybrid arrangements.

Qualifications

• 3+ years of data engineering or related experience.
• Strong Spark experience with large-scale data processing.
• Experience with ETL/ELT pipelines and workflow orchestration.

Responsibilities

• Build and maintain data transformation pipelines using Java Spark.
• Develop and optimize large-scale CPU-intensive data processing with Apache Spark.
• Orchestrate workflows using Airflow.
• Implement data quality checks and monitoring for pipelines.
• Support schema evolution, backfills, and incremental processing.
• Ensure pipelines meet SLAs for freshness, reliability, and performance.
• Expertise/working knowledge in Spark and HBase (semantic layer, virtual datasets).

Skills

Spark

Java

Airflow

Data quality

Metadata & lineage

Tools

HBase

OpenTable format/Iceberg

Apache Arrow

AWS

Kubernetes

Job description

Key Responsibilities

• Build and maintain data transformation pipelines using java Spark
• Develop and optimize large-scale/CPU intensive data processing using Apache Spark
• Orchestrate workflows using Airflow
• Implement data quality checks, testing, and monitoring for pipeline. Good to have exposer into managing metadata, cataloguing, and lineage
• Support schema evolution, backfills, and incremental processing
• Ensure pipelines meet SLAs for freshness, reliability, and performance
• Expertise/working knowledge in Spark and HBase(semantic layer, virtual datasets, Reflections)

Strong hands-on experience with HBase

Nice-to-Have Skills

• OpenTable format/Iceberg ,Apache Arrow
• CDC-based analytics pipelines
• Cloud platforms (AWS)
• Kubernetes-based data platforms

Experience- Min 3 years

Location - Pune