Lead Cloudera Streaming Architect

Compunnel, Inc.; Oregon (WI); Remote; USD 120.000 - 150.000; Vollzeit

Zusammenfassung

A leading data solutions firm is seeking a Lead Cloudera Consultant (Solution Architect) to design and optimize real-time streaming pipelines. The role demands extensive experience with Cloudera Data Platform components such as NiFi and Kafka to implement ingestion, transformation, and integration pipelines. Candidates should have over 8 years in data engineering, with a focus on high-volume systems. This position offers a unique opportunity to mentor teams while driving architectural decisions in a cutting-edge streaming ecosystem.

Qualifikationen

• 8+ years of experience in data engineering or streaming system development.
• 3–5+ years of hands-on experience with Cloudera CDP/CDF streaming components.
• Experience with high-volume industries such as financial services, healthcare, or telecom is preferred.

Aufgaben

• Architect and build end-to-end real-time pipelines using NiFi, Kafka, and Flink.
• Develop advanced NiFi flows including JDBC services and schema registry integration.
• Collaborate with engineering teams to optimize deployments and configurations.

Kenntnisse

NiFi

Kafka

Flink

SQL

Kudu

CDC patterns

Snowflake ingestion

Jobbeschreibung

The Lead Cloudera Consultant (Solution Architect) will design, build, and optimize real-time streaming pipelines using the full Cloudera Data Platform (CDP) streaming stack.

This role requires deep hands-on expertise with NiFi, Kafka, Flink, Kudu/Impala, and SQL Stream Builder (SSB) to deliver mission-critical, high-volume Data-in-Motion solutions.

The consultant will architect and implement real-time ingestion, transformation, CDC patterns, and integration pipelines, while guiding engineering teams on best practices across the streaming ecosystem.

Key Responsibilities

• Architect and build end-to-end real-time pipelines using NiFi, Kafka, Flink, Kudu/Impala, and SSB.
• Own architectural decisions, design patterns, schema evolution strategies, state management, and exactly-once delivery.
• Develop advanced NiFi flows including JDBC/DBCP services, record processors, stateful processors, schema registry integration, and high-volume ingestion patterns.
• Build and optimize Flink SQL and DataStream API jobs including event-time windows, watermarks, checkpointing, and state backends.
• Design and tune Kudu tables including primary keys, partition strategies, distribution, upserts, deletes, and merges.
• Build, deploy, and monitor streaming SQL jobs using SQL Stream Builder (SSB).
• Deliver core real-time use cases including NiFi?Snowflake?Impala/Kudu ingestion, Kafka?Flink streaming, Flink?Kafka with exactly-once semantics, and CDC ingestion.
• Tune NiFi, Kafka, and Flink clusters to optimize throughput, reliability, and latency.
• Implement schema governance, back-pressure controls, error-handling frameworks, and replay strategies.
• Collaborate with platform engineering teams to optimize deployments and configurations.
• Provide architectural guidance, documentation, and mentorship to engineering teams.

Required Qualifications

• 8+ years of experience in data engineering or streaming system development.
• 3–5+ years of hands-on experience with Cloudera CDP/CDF streaming components.
• Strong production-grade experience with NiFi, including advanced processors, controller services, CDC patterns, and Snowflake/Kudu ingestion.
• Hands-on expertise with Kafka including topic design, consumer groups, schema registry, partition strategies, and high-volume streaming.
• Strong proficiency with Flink SQL and DataStream API including event-time processing, windows, watermarks, checkpointing, and exactly-once semantics.
• Experience with Kudu table design, schema changes, upserts, deletes, merges, and Impala querying.
• Strong knowledge of SQL Stream Builder, job creation, connectors, and Flink SQL job deployment in CDP.
• Experience with CDC ingestion using NiFi, Flink CDC, or SSB including late events, schema evolution, deletes, and incremental key management.
• Strong SQL and distributed systems fundamentals.
• Experience in high-volume industries such as financial services, healthcare, or telecom is preferred.

Preferred Qualifications

• Experience running NiFi, Kafka, or Flink on Kubernetes.
• Snowflake ingestion experience including staging and COPY INTO patterns.
• Experience with Debezium for CDC.
• Experience with CI/CD for data pipelines.
• Security knowledge including Kerberos, Ranger, and Atlas.