Senior Data Engineer: Real-Time Streaming Architect

Avenue Code; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

Avenue Code is seeking a Senior Data Engineer to design and operate large-scale real-time streaming platforms powering analytics and data-driven products. You will collaborate across platform, analytics, and engineering teams to deliver scalable data solutions and influence architectural decisions.

Responsibilities include building streaming pipelines with Kafka and Flink, implementing Snowpipe Streaming for Snowflake, and optimizing data models using Dynamic Tables, Streams, and Tasks.

Qualifications

• 5+ years of experience in Data Engineering or Software Engineering.
• Production experience with Kafka-based streaming systems.
• Hands-on experience with Apache Flink.
• Solid Snowflake experience including Snowpipe Streaming, Dynamic Tables, Streams and Tasks.
• Experience building semantic layers or analytical cubes with Cube.dev and/or MicroStrategy.
• Strong Java and/or Python; advanced SQL for distributed queries.
• Experience operating production data pipelines with monitoring and incident response.
• Excellent communication and cross-team collaboration.

Responsibilities

• Design, build, and operate real-time streaming data pipelines using Kafka and Flink.
• Develop production-grade streaming apps for large event volumes with low latency.
• Build and maintain Snowpipe Streaming integrations for continuous ingestion into Snowflake.
• Develop and optimize Snowflake data models using Dynamic Tables, Streams, and Tasks.
• Design and maintain semantic layers and analytical models using Cube.dev and MicroStrategy.
• Address data skew with bucketing, salting, repartitioning, and adaptive partitioning.
• Manage Apache Iceberg tables, including partitioning, compaction, and snapshots.
• Collaborate with platform and analytics teams to optimize real-time analytics and reporting.
• Tune streaming jobs and distributed queries to balance performance and cost.
• Troubleshoot ingestion bottlenecks and distributed processing issues.
• Contribute to shared platform components, reviews, and documentation.
• Participate in architecture discussions and best-practice definition.

Skills

Kafka

Flink

Java

Python

SQL

Streaming systems

Data modeling

Tools

Snowflake

Snowpipe Streaming

Iceberg

Cube.dev

MicroStrategy

Snowflake Dynamic Tables

Snowflake Tasks

Job description

Avenue Code is seeking a Senior Data Engineer to design and operate large-scale real-time streaming platforms powering analytics and data-driven products. You will collaborate across platform, analytics, and engineering teams to deliver scalable data solutions and influence architectural decisions.

Responsibilities include building streaming pipelines with Kafka and Flink, implementing Snowpipe Streaming for Snowflake, and optimizing data models using Dynamic Tables, Streams, and Tasks.