Kafka Data Engineer

Veriipro; Pennington (NJ); On-site; USD 150,000 - 190,000; Full time

Job summary

Veriipro is seeking an experienced Kafka/Spark Data Engineer to design and implement real-time data processing solutions. You will work with Kafka, Spark, Redis, Python, and cloud-based Big Data platforms to build scalable streaming architectures.

You will develop streaming applications, integrate NoSQL stores like MongoDB or HBase, and collaborate on data visualization with Tableau. Strong distributed systems knowledge and SQL/NoSQL skills are essential.

Qualifications

• 9+ years of overall IT/industry experience, with strong Big Data and real-time processing experience.
• Bachelor's or Master's degree in Computer Science, Engineering, Science, or related field.
• Strong hands-on experience with Kafka and real-time streaming.
• Experience with Spark, Redis, and Python.
• Strong understanding of distributed systems, data partitioning, and fault-tolerant architectures.
• Experience with NoSQL databases and cloud platforms.
• Strong problem-solving, analytical, and communication skills.

Responsibilities

• Design, develop, and support real-time data streaming and Big Data solutions.
• Develop streaming applications using Apache Kafka, KStreams, and KTables.
• Work with Apache Spark for large-scale data processing.
• Develop applications using Python, Scala, Java, and Bash/Shell scripting.
• Work with NoSQL databases such as Redis, MongoDB, and HBase.
• Design scalable, distributed, and fault-tolerant data architectures.
• Implement data integration, security, and authentication solutions, including Kerberos.
• Work with streaming technologies such as Flink or Storm.
• Support cloud-based data solutions across AWS, Azure, or GCP.
• Apply data analysis, statistical methods, and machine learning techniques where required.
• Create and support data visualizations using Tableau.
• Troubleshoot business-critical applications and provide effective technical solutions.

Skills

Kafka

Spark

Redis

Python

NoSQL databases

Bash/Shell scripting

Distributed systems

Real-time processing

Cloud platforms

Kafka Streams

KTables

KStreams

Tableau

MongoDB

HBase

Education

Bachelor's or Master's degree in Computer Science, Engineering, Science, or related field

Tools

KStreams

KTables

MongoDB

HBase

Tableau

Job description

We are seeking an experienced Kafka/Spark Data Engineer with strong expertise in real-time data processing, streaming technologies, and distributed systems. The role will focus on Kafka, Spark, Redis, Python, and cloud-based Big Data solutions.

Roles and Responsibilities

• Design, develop, and support real-time data streaming and Big Data solutions.
• Develop streaming applications using Apache Kafka, KStreams, and KTables.
• Work with Apache Spark for large-scale data processing.
• Develop applications using Python, Scala, Java, and Bash/Shell scripting.
• Work with NoSQL databases such as Redis, MongoDB, and HBase.
• Design scalable, distributed, and fault-tolerant data architectures.
• Implement data integration, security, and authentication solutions, including Kerberos.
• Work with streaming technologies such as Flink or Storm.
• Support cloud-based data solutions across AWS, Azure, or GCP.
• Apply data analysis, statistical methods, and machine learning techniques where required.
• Create and support data visualizations using Tableau.
• Troubleshoot business-critical applications and provide effective technical solutions.

Required Qualifications

• 9+ years of overall IT/industry experience, with strong Big Data and real-time processing experience.
• Bachelor's or Master's degree in Computer Science, Engineering, Science, or related field.
• Strong hands-on experience with Kafka and real-time streaming.
• Experience with Spark, Redis, and Python.
• Strong understanding of distributed systems, data partitioning, and fault-tolerant architectures.
• Experience with NoSQL databases and cloud platforms.
• Strong problem-solving, analytical, and communication skills.

Preferred Qualifications

• Experience with Flink, Storm, Scala, or Java.
• Knowledge of machine learning and predictive analytics.
• Experience with Tableau and real-time data visualization.