Senior Kafka & Flink Streaming Engineer

Zone IT Solutions Pty Ltd; Melbourne; On-site; AUD 120,000 - 180,000; Full time

Job summary

Zone IT Solutions Pty Ltd in Melbourne is seeking an experienced Integration Engineer to design and implement data-driven, event-based integration patterns.

You will develop Java/Spring Boot connectors, manage Kafka pipelines on Confluent Cloud/NGIP, and build Flink SQL streaming jobs for transformation, validation, and routing. Strong knowledge of Schema Registry and API integrations via Kong Gateway is required.

Qualifications

• Must develop custom integration code using Java / Spring Boot for source system connectivity.
• Must build and maintain Kafka producers/consumers and event-driven services on Confluent Cloud/NGIP.
• Must develop and maintain Flink SQL / stream processing logic for transformation, enrichment, validation, routing, and de-duplication.
• Must implement integration with Schema Registry, topic schemas, and message validation controls; support HTTP/API-based integrations via Kong Gateway.
• Must configure and support HTTP Sink delivery patterns, payload mapping, retries, and failure handling.

Responsibilities

• Develop custom integration code using Java / Spring Boot for source system connectivity where managed connectors are not applicable.
• Build and maintain Kafka producers / consumers, topic integration logic, and event-driven services on Confluent Cloud / NGIP.
• Develop and maintain Flink SQL / stream processing logic for transformation, enrichment, validation, routing, and de-duplication.
• Implement integration with Schema Registry, topic schemas, and message validation controls.
• Support HTTP/API-based integrations through Kong Gateway for NGIP to target application communication
• Configure and support HTTP Sink delivery patterns, payload mapping, retries, and failure handling.
• Perform unit / component testing for custom code, Flink jobs, and interface logic.
• Support SIT, pipeline testing, defect fixing, replay, and root-cause analysis during test execution
• Support E2E testing with downstream teams and resolve integration issues across connectors, transformations, APIs, and Kafka topics
• Work with GitLab CI/CD pipelines for build, deployment, regression automation, and release support

Skills

Java

Spring Boot

Kafka

Confluent Cloud

NGIP

Flink SQL

Schema Registry

HTTP/API integration

Kong Gateway

HTTP Sink

Payload mapping

Retries

Failure handling

Tools

GitLab CI/CD

Kong Gateway

Job description

Must Have:

· Develop custom integration code using Java / Spring Boot for source system connectivity where managed connectors are not applicable.

· Build and maintain Kafka producers / consumers, topic integration logic, and event-driven services on Confluent Cloud / NGIP.

· Develop and maintain Flink SQL / stream processing logic for transformation, enrichment, validation, routing, and de-duplication.

· Implement integration with Schema Registry, topic schemas, and message validation controls. · Support HTTP/API-based integrations through Kong Gateway for NGIP to target application communication.

· Configure and support HTTP Sink delivery patterns, payload mapping, retries, and failure handling.

Good To Have:

· Experience with stream governance, RBAC, and tiered storage

· Background in data engineering, ETL, or microservices architecture

· Familiarity with traditional messaging systems and data warehouses

· Experience in Agile methodologies and DevOps practice.

SN Responsibility of / Expectations from the Role

1 Develop custom integration code using Java / Spring Boot for source system connectivity where managed connectors are not applicable

2 Build and maintain Kafka producers / consumers, topic integration logic, and event-driven services on Confluent Cloud / NGIP.

3 Develop and maintain Flink SQL / stream processing logic for transformation, enrichment, validation, routing, and de-duplication.

4 Implement integration with Schema Registry, topic schemas, and message validation controls..

5 Support HTTP/API-based integrations through Kong Gateway for NGIP to target application communication

6 Configure and support HTTP Sink delivery patterns, payload mapping, retries, and failure handling

7 Perform unit / component testing for custom code, Flink jobs, and interface logic.

8 Support SIT, pipeline testing, defect fixing, replay, and root-cause analysis during test execution

9 Support E2E testing with downstream teams and resolve integration issues across connectors, transformations, APIs, and Kafka topics

10 Work with GitLab CI/CD pipelines for build, deployment, regression automation, and release support