Senior Real-Time Data Engineer: Kafka & Databricks

Intellias

Fort Snelling Unorganized Territory (MN)

On-site

USD 120,000 - 180,000

Full time

17 hours ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Intellias is seeking an experienced Senior Data Engineer specializing in Apache Kafka Streams and Databricks to design, develop, and optimize scalable, real-time data processing applications and cloud-based data solutions for a medical device client.

You will design and implement streaming ETL/ELT pipelines, build Databricks workloads with PySpark, and ensure robust fault-tolerant data processing at scale. Strong Java, Python, and SQL skills are required.

Qualifications

  • 5+ years of experience in software/data engineering or a related technical field.
  • Strong hands-on experience developing Java-based Apache Kafka Streams applications.
  • Experience with Confluent Kafka, including topics, partitions, consumer groups, and Schema Registry.
  • Deep understanding of Kafka Streams concepts (KStream, KTable, GlobalKTable, joins, aggregations, windowing, state stores).
  • Hands-on experience building Databricks applications using Python, PySpark, and Spark Structured Streaming.
  • Proficiency in Java, Python, PySpark, and SQL.
  • Experience developing real-time Kafka consumers and integrating streaming data into Databricks.
  • Experience building scalable ETL/ELT pipelines using Databricks and Delta Lake.
  • Strong understanding of Kafka offset management, checkpointing, fault tolerance, and error handling.
  • Experience using Gradle for Java application builds and dependency management.
  • Familiarity with Git, CI/CD pipelines, application monitoring, and performance optimization.
  • Knowledge of Apache Iceberg, Unity Catalog, and modern Lakehouse architectures.

Responsibilities

  • Design, develop, and maintain Java-based Kafka Streams applications for real-time data processing.
  • Build and optimize Databricks applications using Python, PySpark, and Spark Structured Streaming.
  • Develop Kafka consumers within Databricks to ingest and process real-time data from Confluent Kafka topics.
  • Design and implement scalable ETL/ELT pipelines using Databricks and Delta Lake.
  • Implement Kafka Streams processing capabilities, including joins, aggregations, windowing, and stateful processing.
  • Ensure reliable data processing through effective offset management, checkpointing, fault tolerance, and error handling.
  • Optimize application performance, scalability, and data processing efficiency.
  • Support application development, dependency management, and deployment using Git, Gradle, and CI/CD pipelines.
  • Collaborate with engineering teams to develop reliable, cloud-based data solutions.
  • Apply best practices for data governance, monitoring, and Lakehouse architecture.

Skills

Java
Python
SQL
Kafka Streams
Databricks

Tools

Databricks
PySpark
Spark Structured Streaming
Gradle
Git
CI/CD
Confluent Kafka
Kafka Streams
Iceberg
Unity Catalog
Delta Lake

Job description

Intellias is seeking an experienced Senior Data Engineer specializing in Apache Kafka Streams and Databricks to design, develop, and optimize scalable, real-time data processing applications and cloud-based data solutions for a medical device client.

You will design and implement streaming ETL/ELT pipelines, build Databricks workloads with PySpark, and ensure robust fault-tolerant data processing at scale. Strong Java, Python, and SQL skills are required.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer-Kafka
Senior Data Engineer-Kafka

Intellias • Fort Snelling Unorganized Territory (MN)

On-site
USD 120,000 - 180,000
Remote Real-Time Data Engineer | Kafka & Databricks
Remote Real-Time Data Engineer | Kafka & Databricks

Central Strategies, LLC • Washington

Remote
USD 110,000 - 150,000
Senior Data Engineer
Senior Data Engineer

The Value Maximizer • South Carolina

On-site
USD 120,000 - 150,000
Senior Data Engineer
Senior Data Engineer

Veriipro • Atlanta (GA)

On-site
USD 140,000 - 180,000
Senior Real-Time Kafka & Spark Data Engineer
Senior Real-Time Kafka & Spark Data Engineer

Veriipro • Pennington (NJ)

On-site
USD 150,000 - 190,000
Databricks Data Engineer Lead: Streaming & ETL Expert
Databricks Data Engineer Lead: Streaming & ETL Expert

August Schell • Rockville (MD)

On-site
USD 120,000 - 170,000
Kafka Data Engineer
Kafka Data Engineer

Veriipro • Pennington (NJ)

On-site
USD 150,000 - 190,000
Real-Time Data Engineer - Kafka & Streaming Architect
Real-Time Data Engineer - Kafka & Streaming Architect

The Value Maximizer • South Carolina

On-site
USD 120,000 - 150,000
Senior Data Engineer - Real-Time Pipelines (Spark/Kafka)
Senior Data Engineer - Real-Time Pipelines (Spark/Kafka)

UST • Sunnyvale (CA)

On-site
USD 72,000 - 108,000
Medical Insurance
401(k) with employer match
Paid time off
Data Engineer
Data Engineer

GSquared Group • Atlanta (GA)

On-site
USD 120,000 - 160,000