Hadoop Developer

Veriipro; Charlotte (NC); On-site; USD 90,000 - 120,000; Full time

Job summary

Veriipro is seeking a skilled professional to integrate Kafka with Hadoop workflows. You will configure and deploy Kafka clusters, develop Spark jobs in Scala or Java, and ensure efficient data processing.

The ideal candidate should have strong Kafka and Spark expertise, as well as experience with performance tuning in Hadoop. Collaboration in an Agile team setting is essential for this role.

Qualifications

• Experience with Kafka configurations, monitoring, and deployments.
• Working knowledge of Spark and Scala/Java.
• Understanding Kafka concepts: topics, partitions, offsets.

Responsibilities

• Integrate Kafka with Hadoop workflows.
• Configure, monitor, and deploy Kafka clusters.
• Develop and optimize Spark jobs in Scala/Java.

Skills

Kafka configurations

Spark and Scala/Java

Kafka brokers knowledge

HDFS knowledge

Shell scripting

Tools

Cloudera platform (CDP)

Git/Bitbucket

Job description

Required Skills

• Kafka configurations, monitoring, and deployments
• Experience with Spark and Scala/Java
• Working knowledge of Kafka brokers, producers, and consumers
• Understanding Kafka concepts: topics, partitions, offsets
• HDFS, MapReduce, and Hadoop ecosystem knowledge
• Data storage and processing techniques in Hadoop
• Performance tuning of Hadoop and related components
• Shell scripting
• Experience with Cloudera platform (CDP)

Preferred Skills

• Agile methodology experience (Jira, Confluence)
• CI/CD deployment experience
• Experience with Git/Bitbucket
• Metadata-driven ETL knowledge
• Banking domain knowledge

Roles & Responsibilities

• Integrate Kafka with Hadoop workflows
• Configure, monitor, and deploy Kafka clusters
• Develop and optimize Spark jobs in Scala/Java
• Ensure efficient data storage and processing in Hadoop
• Perform performance tuning and troubleshooting
• Support CI/CD pipelines for ETL processes
• Work collaboratively in an Agile team environment