Hadoop Developer: Kafka & Spark Data Engineer

Veriipro; Charlotte (NC); On-site; Full time

Job description

Required Skills

• Kafka configurations, monitoring, and deployments
• Experience with Spark and Scala/Java
• Working knowledge of Kafka brokers, producers, and consumers
• Understanding Kafka concepts: topics, partitions, offsets
• HDFS, MapReduce, and Hadoop ecosystem knowledge
• Data storage and processing techniques in Hadoop
• Performance tuning of Hadoop and related components
• Shell scripting
• Experience with Cloudera platform (CDP)

Preferred Skills

• Agile methodology experience (Jira, Confluence)
• CI/CD deployment experience
• Experience with Git/Bitbucket
• Metadata-driven ETL knowledge
• Banking domain knowledge

Roles & Responsibilities

• Integrate Kafka with Hadoop workflows
• Configure, monitor, and deploy Kafka clusters
• Develop and optimize Spark jobs in Scala/Java
• Ensure efficient data storage and processing in Hadoop
• Perform performance tuning and troubleshooting
• Support CI/CD pipelines for ETL processes
• Work collaboratively in an Agile team environment