Data Engineer With Java + Spark

Wissen Technology; Pune District; Hybrid; INR 1,100,000 - 1,700,000; Full time

Job summary

Wissen Technology in Pune seeks a data engineer to build and optimize large-scale data pipelines using Apache Spark and HBase. You will implement data quality checks, manage schema evolution, and ensure SLAs for freshness and reliability.

The role involves developing Spark-based transformations, orchestrating workflows with Airflow, and collaborating on metadata cataloging and lineage. Familiarity with Git, CI/CD, and cloud basics is a plus.

Qualifications

• Strong hands-on experience with Apache Spark and HBase.
• Experience with Airflow for workflow orchestration.
• Proficiency in Java and Git-based development with CI/CD.
• Familiarity with data catalogs and lineage tools such as OpenLineage, DataHub or Apache Polaris.

Responsibilities

• Build and maintain data transformation pipelines using Java Spark.
• Develop and optimize large-scale data processing.
• Orchestrate workflows using Airflow.
• Implement data quality checks, testing, and monitoring.
• Support schema evolution, backfills, and incremental processing.
• Ensure pipelines meet SLAs for freshness, reliability and performance.
• Expertise with Spark and HBase; understand lakehouse concepts and lineage.

Tools

Apache Spark

HBase

Java

Airflow

Git

CI/CD

OpenLineage

DataHub

Apache Polaris

OpenTable/Iceberg

Apache Arrow

AWS

Kubernetes

Job description

Key Responsibilities

• Build and maintain data transformation pipelines using java Spark
• Develop and optimize large-scale/CPU intensive data processing using Apache Spark
• Orchestrate workflows using Airflow
• Implement data quality checks, testing, and monitoring for pipeline. Good to have exposer into managing metadata, cataloguing, and lineage
• Support schema evolution, backfills, and incremental processing
• Ensure pipelines meet SLAs for freshness, reliability, and performance
• Expertise/working knowledge in Spark and HBase(semantic layer, virtual datasets, Reflections)

Required Skills & Qualifications

• Strong hands-on experience with
• HBase
• Apache Spark
• Experience with HBase or similar lakehouse query engines
• Airflow
• Understanding of data catalogs and lineage (e.g., OpenLineage, DataHub, Apache Polaris , openlineage)
• Proficiency in Java
• Experience with Git-based development and CI/CD

Nice-to-Have Skills

• OpenTable format/Iceberg ,Apache Arrow
• CDC-based analytics pipelines
• Cloud platforms (AWS)
• Kubernetes-based data platforms

Experience- Min 3 years

Location - Pune