Intellias is seeking an experienced Senior Data Engineer specializing in Apache Kafka Streams and Databricks to support one of our leading medical device clients. In this role, you will help design, develop, and optimize scalable, real-time data processing applications and cloud-based data solutions. You will work with modern streaming technologies, distributed data architectures, and advanced analytics platforms to support critical business and technology initiatives within the medical device industry. This is an excellent opportunity for an engineer who enjoys solving complex data challenges, building high-performance applications, and working in a collaborative, innovative environment.
Responsibilities
- Design, develop, and maintain Java-based Kafka Streams applications for real-time data processing.
- Build and optimize Databricks applications using Python, PySpark, and Spark Structured Streaming.
- Develop Kafka consumers within Databricks to ingest and process real-time data from Confluent Kafka topics.
- Design and implement scalable ETL/ELT pipelines using Databricks and Delta Lake.
- Implement Kafka Streams processing capabilities, including joins, aggregations, windowing, and stateful processing.
- Ensure reliable data processing through effective offset management, checkpointing, fault tolerance, and error handling.
- Optimize application performance, scalability, and data processing efficiency.
- Support application development, dependency management, and deployment using Git, Gradle, and CI/CD pipelines.
- Collaborate with engineering teams to develop reliable, cloud-based data solutions.
- Apply best practices for data governance, monitoring, and Lakehouse architecture.
Qualifications
- 5+ years of experience in software engineering, data engineering, or a related technical field.
- Strong hands-on experience developing Java-based Apache Kafka Streams applications.
- Experience with Confluent Kafka, including topics, partitions, consumer groups, and Schema Registry.
- Strong understanding of Kafka Streams concepts, including KStream, KTable, GlobalKTable, joins, aggregations, windowing, and state stores.
- Hands-on experience building Databricks applications using Python, PySpark, and Spark Structured Streaming.
- Proficiency in Java, Python, PySpark, and SQL.
- Experience developing real-time Kafka consumers and integrating streaming data into Databricks.
- Experience building scalable ETL/ELT pipelines using Databricks and Delta Lake.
- Strong understanding of Kafka offset management, checkpointing, fault tolerance, and error handling.
- Experience using Gradle for Java application builds and dependency management.
- Familiarity with Git, CI/CD pipelines, application monitoring, and performance optimization.
- Knowledge of Apache Iceberg, Unity Catalog, and modern Lakehouse architectures.
Required Skills
- Confluent Kafka (topics, partitions, consumer groups, Schema Registry).
- Kafka Streams concepts (KStream, KTable, GlobalKTable, joins, aggregations, windowing, state stores).
- Databricks applications using Python, PySpark, and Spark Structured Streaming.
- Java, Python, PySpark, SQL.
- Real-time Kafka consumers and streaming data integration into Databricks.
- ETL/ELT pipelines using Databricks and Delta Lake.
- Kafka offset management, checkpointing, fault tolerance, error handling.
- Gradle for Java application builds and dependency management.
- Git, CI/CD pipelines, application monitoring, performance optimization.
- Apache Iceberg, Unity Catalog, modern Lakehouse architectures.
Preferred Skills
- Experience with AWS cloud services and cloud-based data engineering architectures.
- Experience designing and optimizing distributed data processing systems.
- Familiarity with enterprise data governance and data quality practices.
- Previous experience supporting healthcare, medical device, or other regulated industries.