DATA Engineer (JAVA SPARK)

Wissen Technology; Pune District; Hybrid; INR 1,400,000 - 2,200,000; Full time

Job summary

Wissen Technology Pune is seeking a data engineering expert to build and optimize large-scale Spark pipelines. You will manage data quality, lineage, and metadata while ensuring on-time delivery and high performance.

The role involves working with HBase, Spark, and Airflow, with some exposure to streaming and cataloging tasks. The position requires Java proficiency, Git-based development, and CI/CD familiarity, with 3 days in office as per the listing.

Qualifications

• Strong hands-on experience with HBase.
• Apache Spark experience.
• Experience with HBase or similar lakehouse query engines.
• Airflow experience.
• Understanding of data catalogs and lineage (OpenLineage, DataHub, Apache Polaris, openlineage).
• Proficiency in Java.
• Experience with Git-based development and CI/CD.

Responsibilities

• Build and maintain data transformation pipelines using java Spark
• Develop and optimize large-scale/CPU intensive data processing using Apache Spark
• Orchestrate workflows using Airflow
• Implement data quality checks, testing, and monitoring for pipeline.
• Good to have exposure into managing metadata, cataloguing, and lineage
• Hbase r t schema evolution, backfills, and incremental processing
• Ensure pipelines meet SLAs for freshness, reliability, and performance

Skills

Apache Spark

HBase

Java

Airflow

Data catalogs & lineage

OpenLineage/DataHub/OpenSource lineage

CI/CD

Git-based development

Tools

Git

CI/CD tooling

Job description

JOB DESCRIPTION:

Location: Pune

Mode of Work : 3 days from Office

Key Responsibilities

• Build and maintain data transformation pipelines using java Spark
• Develop and optimize large-scale/CPU intensive data processing using Apache Spark
• Orchestrate workflows using Airflow
• Implement data quality checks, testing, and monitoring for pipeline. Good to have exposer into managing metadata, cataloguing, and lineage
• Hbasert schema evolution, backfills, and incremental processing
• Ensure pipelines meet SLAs for freshness, reliability, and performance
• Expertise/working knowledge in Spark and HBase(semantic layer, virtual datasets, Reflections)

Required Skills & Qualifications

• Strong hands-on experience with HBase
• Apache Spark
• Experience with HBase or similar lakehouse query engines
• Airflow
• Understanding of data catalogs and lineage (e.g., OpenLineage, DataHub, Apache Polaris , openlineage)
• Proficiency in Java
• Experience with Git-based development and CI/CD

Nice-to-Have Skills

• OpenTable format/Iceberg ,Apache Arrow
• CDC-based analytics pipelines
• Cloud platforms (AWS)
• Kubernetes-based data platforms