Senior Data Engineer-Kafka

Intellias

Fort Snelling Unorganized Territory (MN)

Presencial

USD 120 000 - 180 000

Tempo integral

há 17 horas
Torna-te num dos primeiros candidatos

Uma candidatura completa num minuto

Currículo e carta de apresentação personalizados, prontos a enviar.

Ultrapassa os filtros ATS

Resumo da oferta

Intellias is seeking an experienced Senior Data Engineer specializing in Apache Kafka Streams and Databricks to design, develop, and optimize scalable, real-time data processing applications and cloud-based data solutions for a medical device client.

You will design and implement streaming ETL/ELT pipelines, build Databricks workloads with PySpark, and ensure robust fault-tolerant data processing at scale. Strong Java, Python, and SQL skills are required.

Qualificações

  • 5+ years of experience in software/data engineering or a related technical field.
  • Strong hands-on experience developing Java-based Apache Kafka Streams applications.
  • Experience with Confluent Kafka, including topics, partitions, consumer groups, and Schema Registry.
  • Deep understanding of Kafka Streams concepts (KStream, KTable, GlobalKTable, joins, aggregations, windowing, state stores).
  • Hands-on experience building Databricks applications using Python, PySpark, and Spark Structured Streaming.
  • Proficiency in Java, Python, PySpark, and SQL.
  • Experience developing real-time Kafka consumers and integrating streaming data into Databricks.
  • Experience building scalable ETL/ELT pipelines using Databricks and Delta Lake.
  • Strong understanding of Kafka offset management, checkpointing, fault tolerance, and error handling.
  • Experience using Gradle for Java application builds and dependency management.
  • Familiarity with Git, CI/CD pipelines, application monitoring, and performance optimization.
  • Knowledge of Apache Iceberg, Unity Catalog, and modern Lakehouse architectures.

Responsabilidades

  • Design, develop, and maintain Java-based Kafka Streams applications for real-time data processing.
  • Build and optimize Databricks applications using Python, PySpark, and Spark Structured Streaming.
  • Develop Kafka consumers within Databricks to ingest and process real-time data from Confluent Kafka topics.
  • Design and implement scalable ETL/ELT pipelines using Databricks and Delta Lake.
  • Implement Kafka Streams processing capabilities, including joins, aggregations, windowing, and stateful processing.
  • Ensure reliable data processing through effective offset management, checkpointing, fault tolerance, and error handling.
  • Optimize application performance, scalability, and data processing efficiency.
  • Support application development, dependency management, and deployment using Git, Gradle, and CI/CD pipelines.
  • Collaborate with engineering teams to develop reliable, cloud-based data solutions.
  • Apply best practices for data governance, monitoring, and Lakehouse architecture.

Conhecimentos

Java
Python
SQL
Kafka Streams
Databricks

Ferramentas

Databricks
PySpark
Spark Structured Streaming
Gradle
Git
CI/CD
Confluent Kafka
Kafka Streams
Iceberg
Unity Catalog
Delta Lake

Descrição da oferta de emprego

Intellias is seeking an experienced Senior Data Engineer specializing in Apache Kafka Streams and Databricks to support one of our leading medical device clients. In this role, you will help design, develop, and optimize scalable, real-time data processing applications and cloud-based data solutions. You will work with modern streaming technologies, distributed data architectures, and advanced analytics platforms to support critical business and technology initiatives within the medical device industry. This is an excellent opportunity for an engineer who enjoys solving complex data challenges, building high-performance applications, and working in a collaborative, innovative environment.

Responsibilities
  • Design, develop, and maintain Java-based Kafka Streams applications for real-time data processing.
  • Build and optimize Databricks applications using Python, PySpark, and Spark Structured Streaming.
  • Develop Kafka consumers within Databricks to ingest and process real-time data from Confluent Kafka topics.
  • Design and implement scalable ETL/ELT pipelines using Databricks and Delta Lake.
  • Implement Kafka Streams processing capabilities, including joins, aggregations, windowing, and stateful processing.
  • Ensure reliable data processing through effective offset management, checkpointing, fault tolerance, and error handling.
  • Optimize application performance, scalability, and data processing efficiency.
  • Support application development, dependency management, and deployment using Git, Gradle, and CI/CD pipelines.
  • Collaborate with engineering teams to develop reliable, cloud-based data solutions.
  • Apply best practices for data governance, monitoring, and Lakehouse architecture.
Qualifications
  • 5+ years of experience in software engineering, data engineering, or a related technical field.
  • Strong hands-on experience developing Java-based Apache Kafka Streams applications.
  • Experience with Confluent Kafka, including topics, partitions, consumer groups, and Schema Registry.
  • Strong understanding of Kafka Streams concepts, including KStream, KTable, GlobalKTable, joins, aggregations, windowing, and state stores.
  • Hands-on experience building Databricks applications using Python, PySpark, and Spark Structured Streaming.
  • Proficiency in Java, Python, PySpark, and SQL.
  • Experience developing real-time Kafka consumers and integrating streaming data into Databricks.
  • Experience building scalable ETL/ELT pipelines using Databricks and Delta Lake.
  • Strong understanding of Kafka offset management, checkpointing, fault tolerance, and error handling.
  • Experience using Gradle for Java application builds and dependency management.
  • Familiarity with Git, CI/CD pipelines, application monitoring, and performance optimization.
  • Knowledge of Apache Iceberg, Unity Catalog, and modern Lakehouse architectures.
Required Skills
  • Confluent Kafka (topics, partitions, consumer groups, Schema Registry).
  • Kafka Streams concepts (KStream, KTable, GlobalKTable, joins, aggregations, windowing, state stores).
  • Databricks applications using Python, PySpark, and Spark Structured Streaming.
  • Java, Python, PySpark, SQL.
  • Real-time Kafka consumers and streaming data integration into Databricks.
  • ETL/ELT pipelines using Databricks and Delta Lake.
  • Kafka offset management, checkpointing, fault tolerance, error handling.
  • Gradle for Java application builds and dependency management.
  • Git, CI/CD pipelines, application monitoring, performance optimization.
  • Apache Iceberg, Unity Catalog, modern Lakehouse architectures.
Preferred Skills
  • Experience with AWS cloud services and cloud-based data engineering architectures.
  • Experience designing and optimizing distributed data processing systems.
  • Familiarity with enterprise data governance and data quality practices.
  • Previous experience supporting healthcare, medical device, or other regulated industries.
Obtém a tua avaliação gratuita e confidencial do currículo.

ou arrasta e larga o ficheiro aqui.

Similar jobs

Ofertas semelhantes que vale a pena comparar

Senior Real-Time Data Engineer: Kafka & Databricks
Senior Real-Time Data Engineer: Kafka & Databricks

Intellias • Fort Snelling Unorganized Territory (MN)

Presencial
USD 120 000 - 180 000
Principal/Lead Data Engineer
Principal/Lead Data Engineer

Accion Labs • EUA

Presencial
USD 180 000 - 260 000
Senior Data Engineer
Senior Data Engineer

The Value Maximizer • South Carolina

Presencial
USD 120 000 - 150 000
Data Engineer
Data Engineer

GSquared Group • Atlanta (GA)

Presencial
USD 120 000 - 160 000
Senior Data Streaming Engineer
Senior Data Streaming Engineer

First Soft Solutions LLC • Woodlawn (MD)

Presencial
USD 140 000 - 190 000
Onsite work 5 days/week
Kafka Data Engineer
Kafka Data Engineer

Veriipro • Pennington (NJ)

Presencial
USD 150 000 - 190 000
Kafka, Redis And Hadoop Developer
Kafka, Redis And Hadoop Developer

Veriipro • Addison (TX)

Presencial
USD 140 000 - 210 000
Senior Confluent Kafka & Flink Streaming Architect
Senior Confluent Kafka & Flink Streaming Architect

TechDigital Group • Columbus (OH)

Presencial
USD 180 000 - 280 000
Scala Developer
Scala Developer

Compunnel Inc. • Greenwood Village (CO)

Presencial
USD 90 000 - 130 000
Senior Data Engineer
Senior Data Engineer

Veriipro • Atlanta (GA)

Presencial
USD 140 000 - 180 000