Data Engineer With Java + Spark

Wissen Technology; Pune District; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Wissen Technology is seeking an experienced data engineer to build and maintain large-scale data pipelines using Java and Apache Spark in Pune. You will orchestrate workflows with Airflow, implement data quality checks, and support schema evolution while ensuring SLAs for freshness and performance.

Familiarity with lakehouse concepts and data catalogs is preferred. The role requires hands-on Spark, HBase, and Java skills, plus experience with Git-based development and CI/CD.

Qualifications

• Strong hands-on experience with HBase.
• Proficient in Apache Spark and Java development.
• Experience with Spark-based lakehouse or similar engines and Airflow.
• Knowledge of data catalogs and lineage tools (OpenLineage, DataHub, Polaris).
• Git-based development and CI/CD practices.

Responsibilities

• Build and maintain data transformation pipelines using Java Spark.
• Develop and optimize large-scale data processing using Spark.
• Orchestrate workflows with Airflow.
• Implement data quality checks, testing, and monitoring for pipelines.
• Support schema evolution, backfills, and incremental processing.
• Ensure pipelines meet SLAs for freshness, reliability, and performance.
• Expertise in Spark and HBase and related lakehouse concepts.

Skills

HBase

Apache Spark

Java

Airflow

OpenLineage/DataHub/Polaris

Git

CI/CD

Tools

HBase

Iceberg

Kubernetes

AWS

Job description

Key Responsibilities

• Build and maintain data transformation pipelines using java Spark
• Develop and optimize large-scale/CPU intensive data processing using Apache Spark
• Orchestrate workflows using Airflow
• Implement data quality checks, testing, and monitoring for pipeline. Good to have exposer into managing metadata, cataloguing, and lineage
• Support schema evolution, backfills, and incremental processing
• Ensure pipelines meet SLAs for freshness, reliability, and performance
• Expertise/working knowledge in Spark and HBase(semantic layer, virtual datasets, Reflections)

Required Skills & Qualifications

• Strong hands‑on experience with HBase
• Apache Spark
• Experience with HBase or similar lakehouse query engines
• Airflow
• Understanding of data catalogs and lineage (e.g., OpenLineage, DataHub, Apache Polaris , openlineage)
• Proficiency in Java
• Experience with Git-based development and CI/CD

Nice-to-Have Skills

• OpenTable format/Iceberg ,Apache Arrow
• CDC-based analytics pipelines
• Cloud platforms (AWS)
• Kubernetes-based data platforms

Experience- Min 3 years

Location - Pune