Java Spark

Wissen Technology; Pune District; Hybrid; INR 1,200,000 - 1,800,000; Full time

Job summary

Wissen Technology in Pune invites a data engineer to build and optimize large-scale data pipelines using Spark and Java. You will design transformations, implement quality checks, and ensure readiness for analytics workloads.

You will work with Airflow, HBase and data catalog concepts, contribute to metadata management, and help with schema evolution and backfills in a fast-paced environment.

Qualifications

• Hands-on experience with Spark and HBase
• Proficiency in Java and Airflow
• Experience with Git-based development and CI/CD
• Understanding of data catalogs and lineage (OpenLineage, DataHub, Polaris)

Responsibilities

• Build and maintain data pipelines with Java Spark
• Develop and optimize large-scale Spark processing
• Orchestrate workflows with Airflow
• Implement data quality checks, testing, and monitoring
• Support schema evolution, backfills, and incremental processing
• Ensure pipelines meet SLA for freshness, reliability, and performance
• Leverage Spark and HBase for semantic layer and datasets

Skills

Apache Spark

HBase

Java

Airflow

Git & CI/CD

Data catalogs

OpenLineage

DataHub

Polaris

Iceberg

OpenTable format

AWS

Tools

OpenLineage

DataHub

Polaris

Iceberg

OpenTable format

Apache Arrow

AWS

Kubernetes

Job description

Key Responsibilities
Build and maintain data transformation pipelines using java Spark
Develop and optimize large-scale/CPU intensive data processing using Apache Spark
Orchestrate workflows using Airflow
Implement data quality checks, testing, and monitoring for pipeline. Good to have exposer into managing metadata, cataloguing, and lineage
Support schema evolution, backfills, and incremental processing
Ensure pipelines meet SLAs for freshness, reliability, and performance
Expertise/working knowledge in Spark and HBase(semantic layer, virtual datasets, Reflections)

Required Skills & Qualifications
Strong hands-on experience with
HBase

Apache Spark
Experience with HBase or similar lakehouse query engines
Airflow
Understanding of data catalogs and lineage (e.g., OpenLineage, DataHub, Apache Polaris , openlineage)
Proficiency in Java
Experience with Git-based development and CI/CD

Nice-to-Have Skills
OpenTable format/Iceberg ,Apache Arrow
CDC-based analytics pipelines
Cloud platforms (AWS)
Kubernetes-based data platforms