Real-Time Big Data Engineer: Kafka, Redis & Hadoop

Veriipro; Addison (TX); On-site; USD 140,000 - 210,000; Full time

Job summary

Veriipro is seeking an experienced Kafka, Redis, and Hadoop Developer to design and maintain scalable data processing solutions in a fast-paced environment. You will leverage Apache Kafka, Redis, Hadoop, PySpark, and Python to deliver real-time ingestion, processing, and analytics.

You will design streaming applications, optimize data pipelines, and collaborate with cross-functional teams to implement secure, fault-tolerant architectures and visual analytics dashboards using Tableau.

Qualifications

• Bachelor's or Master's degree in Science, Engineering, Computer Science, or a related field.
• 9–12 years of overall professional experience in software development and data engineering.
• Strong hands-on experience with Apache Kafka, Redis, Hadoop, and PySpark.
• Proficiency in Python and familiarity with Scala, Java, and Bash/Shell scripting.
• Strong understanding of real-time streaming, distributed computing, and big data architecture.
• Experience with NoSQL databases and cloud-based data processing platforms.
• Knowledge of data security, data integration, and visualization tools such as Tableau.
• Excellent problem-solving, analytical, and communication skills.

Responsibilities

• Design, develop, and maintain real-time data streaming applications using Apache Kafka, including KStreams and KTable.
• Develop scalable big data processing solutions using Hadoop and PySpark.
• Implement and optimize data storage and retrieval using NoSQL databases such as Redis, MongoDB, and HBase.
• Develop data processing and integration pipelines using Python, Scala, Java, and Bash/Shell scripting.
• Build reliable distributed systems with efficient data partitioning, fault tolerance, and high availability.
• Work with streaming technologies such as Apache Flink and Apache Storm.
• Leverage cloud platforms such as AWS, Azure, or GCP for big data processing and deployment.
• Apply data security and integration best practices, including Kerberos authentication.
• Perform data analysis and support predictive modeling using statistical techniques and machine learning algorithms.
• Develop and maintain visual analytics and reporting solutions using Tableau.
• Troubleshoot performance issues, optimize data pipelines, and ensure data quality and reliability.
• Collaborate with cross-functional teams to deliver business-critical data solutions.

Skills

Real-time streaming

Distributed systems

Big data architecture

Problem solving

Communication

Education

Bachelor's or Master's in CS/Engineering

Tools

Apache Kafka

Redis

Hadoop

PySpark

Python

Scala

Java

Bash/Shell

Tableau

AWS

Azure

GCP

Kerberos

Job description

Veriipro is seeking an experienced Kafka, Redis, and Hadoop Developer to design and maintain scalable data processing solutions in a fast-paced environment. You will leverage Apache Kafka, Redis, Hadoop, PySpark, and Python to deliver real-time ingestion, processing, and analytics.

You will design streaming applications, optimize data pipelines, and collaborate with cross-functional teams to implement secure, fault-tolerant architectures and visual analytics dashboards using Tableau.