Data Lead Flink

AI DEFINITIVE INC; Minneapolis (MN); Remote; USD 90,000 - 130,000; Full time

Job summary

AI DEFINITIVE INC is looking for a Java/Apache Flink Engineer to work remotely with a team dedicated to migrating data workflows to a custom Flink framework. The ideal candidate will possess strong skills in Flink, Java, and CI/CD practices, contributing to enhancing data integrations across multiple projects.

Qualifications

• Experience with Flink and real-time data pipeline development required.
• Strong core engineering background.
• CI/CD experience, especially with Azure DevOps, preferred.

Responsibilities

• Develop and manage a custom-built Flink framework.
• Migrate legacy applications to Flink for data movements.
• Integrate more data sources across ongoing projects.

Skills

Flink

Java

Python

CI/CD

GitHub

Cloud

Job description

Title: Java/Apache Flink Engineer (Remote - Must be local to a hub: Minneapolis, Atlanta, Chicago, San Francisco, Dallas)
Contract: 12 Months+

Overview:

This role involves working alongside a full-time engineer and another team member to develop and manage a custom-built Flink framework.

The team is migrating from legacy applications like Informatica PowerExchange to Flink for data movements and workflows.

The Flink framework is used for CDC (Change Data Capture) for MongoDB and SQL Server. A CDC Center of Excellence (COE) has been established within the group.

Description:

The team currently consists of 2 members and needs additional support to increase the velocity of tooling migrations to the Flink framework and to integrate more data sources across 6-7 ongoing projects.

Ideal candidates will have experience with Flink, real-time data pipeline development (Flink SQL, DataStream API), and a strong core engineering background.

The environment is primarily Java, but Python experience is acceptable, especially for Python experts.

Required skills include CI/CD (Azure DevOps preferred), GitHub, and cloud experience (Azure preferred).

Experience in building AI agents is a plus, particularly related to managing Kafka topics, schema registries, and metadata. A potential long-term project involves building a proof of concept with Azure Open AI to generate JSON schemas from Kafka topics. However, this is not an immediate requirement due to funding limitations.