Data Engineer

Smart IT Frame LLC; Columbus (OH); On-site; USD 120,000 - 160,000; Full time

Job summary

Smart IT Frame LLC is seeking an experienced Kafka and Apache Flink Developer to design, develop, and maintain real-time data streaming applications in Columbus, OH. You will build high-throughput pipelines, implement event-time processing, and ensure fault-tolerant execution across distributed systems.

The role requires strong Java expertise, with additional skills in Scala or Python a plus, and hands-on experience with Kafka, Flink, and related ecosystem components.

Qualifications

• Strong hands-on experience with Apache Kafka and Apache Flink.
• Experience developing real-time streaming and event-driven applications.
• Proficiency in Java; Scala or Python experience is an advantage.
• Strong understanding of Kafka producers, consumers, topics, partitions, offsets, and consumer groups.
• Hands-on experience with Flink DataStream API, stateful processing, event-time semantics, and windowing.
• Knowledge of checkpointing, savepoints, fault tolerance, and recovery in Flink.
• Experience with Kafka Connect and Schema Registry.
• Strong understanding of distributed systems, messaging patterns, and stream-processing architecture.
• Experience with SQL, REST APIs, and integration with databases or data platforms.
• Knowledge of monitoring, logging, debugging, and performance tuning.

Responsibilities

• Design, develop, and maintain real-time data streaming pipelines using Apache Kafka and Apache Flink.
• Develop Kafka producers and consumers for reliable, high-throughput event processing.
• Build stream-processing applications using Flink DataStream API, Table API, and SQL.
• Implement event-time processing, windowing, state management, checkpoints, and savepoints in Flink.
• Configure Kafka topics, partitions, replication, consumer groups, retention policies, and offsets.
• Integrate Kafka and Flink with databases, data lakes, APIs, and downstream analytics platforms.
• Implement fault tolerance, exactly-once or at-least-once processing, and recovery mechanisms.
• Monitor and troubleshoot streaming applications, latency, throughput, consumer lag, and processing failures.
• Optimize application performance, resource utilization, and end-to-end data pipeline reliability.
• Implement data serialization and schema management using Avro, Protobuf, or JSON with Schema Registry.
• Work with DevOps teams to deploy and manage streaming applications using Docker, Kubernetes, and CI/CD pipelines.
• Collaborate with architects, data engineers, and business teams to deliver scalable streaming solutions.

Skills

Apache Kafka

Apache Flink

Java

Scala

Python

Event-driven design

Distributed systems

Stream processing

Tools

Docker

Kubernetes

CI/CD

Kafka Connect

Schema Registry

Job description

We are looking for an experienced Kafka and Apache Flink Developer to design, develop, and maintain real-time data streaming applications. The ideal candidate should have strong expertise in event-driven architectures, distributed systems, stream processing, and high-volume data pipelines.

Key Responsibilities

• Design, develop, and maintain real-time data streaming pipelines using Apache Kafka and Apache Flink.
• Develop Kafka producers and consumers for reliable, high-throughput event processing.
• Build stream-processing applications using Flink DataStream API, Table API, and SQL.
• Implement event-time processing, windowing, state management, checkpoints, and savepoints in Flink.
• Configure Kafka topics, partitions, replication, consumer groups, retention policies, and offsets.
• Integrate Kafka and Flink with databases, data lakes, APIs, and downstream analytics platforms.
• Implement fault tolerance, exactly-once or at-least-once processing, and recovery mechanisms.
• Monitor and troubleshoot streaming applications, latency, throughput, consumer lag, and processing failures.
• Optimize application performance, resource utilization, and end-to-end data pipeline reliability.
• Implement data serialization and schema management using Avro, Protobuf, or JSON with Schema Registry.
• Work with DevOps teams to deploy and manage streaming applications using Docker, Kubernetes, and CI/CD pipelines.
• Collaborate with architects, data engineers, and business teams to deliver scalable streaming solutions.

Required Skills

• Strong hands-on experience with Apache Kafka and Apache Flink.
• Experience developing real-time streaming and event-driven applications.
• Proficiency in Java; Scala or Python experience is an advantage.
• Strong understanding of Kafka producers, consumers, topics, partitions, offsets, and consumer groups.
• Hands-on experience with Flink DataStream API, stateful processing, event-time semantics, and windowing.
• Knowledge of checkpointing, savepoints, fault tolerance, and recovery in Flink.
• Experience with Kafka Connect and Schema Registry.
• Strong understanding of distributed systems, messaging patterns, and stream-processing architecture.
• Experience with SQL, REST APIs, and integration with databases or data platforms.
• Knowledge of monitoring, logging, debugging, and performance tuning.