Senior Data Engineer

Incedo Inc.; New Jersey; On-site; USD 120,000 - 170,000; Full time

Job summary

Incedo Inc. is seeking a Senior Data Engineer specializing in event-driven architecture to lead the real-time data backbone of our platform. You will architect streaming pipelines and Kafka-based systems that move data reliably across the company—fueling analytics, ML, and real-time product features.

This senior, hands-on role requires owning architecture, setting standards, and mentoring engineers while driving the platform's streaming direction across the organization.

Qualifications

• 7+ years in data engineering or backend engineering with experience building and operating event-driven or streaming systems at scale.
• Own streaming architecture end-to-end and set technical direction.
• Deep production experience with Apache Kafka (Confluent, MSK, or self-managed).
• Strong patterns in event sourcing, CQRS, pub/sub, CDC.

Responsibilities

• Own architecture of event-driven systems and streaming data pipelines on Apache Kafka.
• Define event models, schemas, and schema-evolution strategy (Avro/Protobuf/JSON).
• Design and lead build-out of producers, consumers, connectors, and stream-processing jobs (Kafka Streams, ksqlDB, Flink, Spark).
• Set topic/partition/consumer-group strategies for scalability and fault tolerance.
• Drive adoption of event-driven patterns and ensure reliability guarantees (exactly-once/at-least-once).
• Lead production operations of Kafka—monitoring, capacity planning, tuning, incident response.
• Architect integrations with source/sink systems via Kafka Connect and Debezium tooling.
• Partner with data/product/platform leads to define event contracts and streaming standards.
• Mentor engineers and raise the technical bar for streaming across the org.

Skills

Kafka

Event-driven arch

Streaming pipelines

Java/Scala

Python

SQL

Tools

Kafka Connect

Debezium

ksqlDB

Flink

Spark Streaming

Schema Registry

Avro/Protobuf

Confluent Platform

Prometheus

Job description

We're looking for a Senior Data Engineer specializing in event-driven architecture to lead the design and operation of the real-time data backbone of our platform. You'll architect streaming pipelines and event-driven systems on Apache Kafka that move data reliably across the company — powering analytics, machine learning, and real-time product features — and set the standards other teams build on.

This is a senior, hands-on role for someone who thinks in events and streams, has scaled Kafka in production, cares deeply about correctness and throughput, and wants to own the architecture and direction of real-time data across the org while mentoring other engineers.

What You'll Do

• Own the architecture of event-driven systems and streaming data pipelines on Apache Kafka.
• Define event models, schemas, and schema-evolution strategy (Avro/Protobuf/JSON, Schema Registry) as company-wide standards.
• Design and lead build-out of producers, consumers, connectors, and stream-processing jobs (Kafka Streams, ksqlDB, Flink, or Spark Streaming).
• Set topic, partitioning, and consumer-group strategies for scalability, ordering, and fault tolerance.
• Drive adoption of patterns like event sourcing, CQRS, change-data-capture (CDC), and the outbox pattern.
• Own reliability guarantees: exactly-once/at-least-once semantics, idempotency, replay, dead-letter handling, and backpressure.
• Lead production operations of Kafka — monitoring, alerting, capacity planning, performance tuning, and incident response.
• Architect integrations with source and sink systems via Kafka Connect and CDC tooling (Debezium and similar).
• Partner with data, product, and platform leads to define event contracts and streaming standards, and influence the platform roadmap.
• Mentor engineers, review designs, and raise the technical bar for streaming across the org.

Required

• 7+ years in data engineering or backend engineering, including substantial experience building and operating event-driven or streaming systems at scale.
• Track record of owning streaming architecture end to end and setting technical direction.
• Deep, production experience with Apache Kafka (Confluent, MSK, or self-managed), including operating it at scale.
• Strong command of event-driven design patterns — event sourcing, CQRS, pub/sub, CDC.
• Proficiency in at least one JVM language (Java/Scala) and/or Python.
• Advanced experience with stream processing (Kafka Streams, ksqlDB, Flink, or Spark Structured Streaming).
• Strong grasp of schemas and serialization (Avro, Protobuf) and Schema Registry, including schema governance.
• Strong SQL and experience across data stores (relational, NoSQL, or data warehouses).

Nice to Have

• Kafka Connect and Debezium (CDC) experience.
• Cloud experience (AWS, GCP, or Azure) and containers/orchestration (Docker, Kubernetes).
• Infrastructure-as-code (Terraform) and CI/CD for streaming pipelines.
• Observability for streaming systems (Prometheus, Grafana, lag monitoring).
• Data lakes/warehouses (Snowflake, BigQuery, Databricks) and lakehouse formats.
• Familiarity with other messaging systems (Pulsar, RabbitMQ, Kinesis).
• Experience mentoring engineers or leading a technical workstream.