Java Spark Data Engineer

Wissen Technology; Pune District; Hybrid; INR 1,300,000 - 2,100,000; Full time

Job summary

Wissen Technology in Pune is hiring a data engineer to build and optimize data transformation pipelines using Java Spark. You will develop large-scale data processing tasks and orchestrate workflows with Airflow to meet freshness and reliability SLAs.

The role requires hands-on experience with Spark and HBase, knowledge of metadata and lineage concepts, and proficiency in Git-based development and CI/CD. Nice-to-have: Iceberg/OpenTable formats and cloud exposure.

Qualifications

• Experience in building data pipelines and transformations.
• Strong ability to optimize large-scale Spark-based workloads.
• Familiarity with data quality checks, testing, and monitoring.

Responsibilities

• Build and maintain data transformation pipelines using java Spark (3+ yrs).
• Develop and optimize large-scale data processing using Apache Spark.
• Orchestrate workflows using Airflow and ensure SLA adherence.

Skills

Data pipeline development

Performance tuning

Problem solving

Team collaboration

Tools

Apache Spark

HBase

Airflow

Java

Git & CI/CD

OpenLineage / DataHub

Job description

Key Responsibilities

• Build and maintain data transformation pipelines using java Spark
• Develop and optimize large-scale/CPU intensive data processing using Apache Spark
• Orchestrate workflows using Airflow
• Implement data quality checks, testing, and monitoring for pipeline. Good to have exposer into managing metadata, cataloguing, and lineage
• Support schema evolution, backfills, and incremental processing
• Ensure pipelines meet SLAs for freshness, reliability, and performance
• Expertise/working knowledge in Spark and HBase(semantic layer, virtual datasets, Reflections)

Strong hands-on experience with HBase

• Apache Spark
• Experience with HBase or similar lakehouse query engines
• Airflow
• Understanding of data catalogs and lineage (e.g., OpenLineage, DataHub, Apache Polaris , openlineage)
• Proficiency in Java
• Experience with Git-based development and CI/CD

Nice-to-Have Skills

• OpenTable format/Iceberg ,Apache Arrow
• CDC-based analytics pipelines
• Cloud platforms (AWS)
• Kubernetes-based data platforms

Experience- Min 3 years

Location - Pune