Big Data Engineer — Kafka, Spark & Cloudera Expert

VeriiPro; Jersey City (NJ); Presencial; USD 120.000 - 180.000; Jornada completa

Descripción de la vacante

VeriiPro is seeking an experienced Big Data Developer to design, build, and optimize data processing pipelines using Kafka, Spark, Scala, SQL, Hadoop, Hive, and Impala. The role focuses on performance tuning and troubleshooting within the Cloudera Big Data ecosystem.

You will work with cross-functional teams to deliver scalable data solutions, ensure data quality, and enable real-time and batch processing for large datasets in a Linux environment.

Formación

• Experience with large-scale data processing and distributed systems.
• Hands-on work with Kafka, Spark, SQL, Hive and Impala in Cloudera ecosystems.
• Strong troubleshooting and performance tuning skills for big data jobs.
• Proficiency in Linux-based environments and batch/workflow scheduling.

Responsabilidades

• Develop, maintain, and optimize big data applications and data processing pipelines.
• Work with Kafka and Apache Spark for real-time and batch data processing.
• Perform data analysis, data wrangling, and processing of large volumes of structured and unstructured data.
• Write and optimize complex SQL queries using Hive and Impala.
• Monitor, troubleshoot, and resolve performance issues in the Cloudera Big Data ecosystem.
• Use Splunk for log analysis, monitoring, and troubleshooting.
• Manage and monitor batch workflows and job scheduling using AutoSys.
• Work in Linux-based environments for development, deployment, and operational support.
• Collaborate with business users and cross-functional teams to deliver high-quality data solutions.
• Ensure data quality, reliability, scalability, and performance of big data applications.

Conocimientos

Kafka

Spark

Scala

SQL

Data analysis

Data wrangling

Performance tuning

Troubleshooting

Linux

Distributed data processing

Herramientas

Hadoop

Hive

Impala

Cloudera

Splunk

AutoSys

Descripción del empleo

VeriiPro is seeking an experienced Big Data Developer to design, build, and optimize data processing pipelines using Kafka, Spark, Scala, SQL, Hadoop, Hive, and Impala. The role focuses on performance tuning and troubleshooting within the Cloudera Big Data ecosystem.

You will work with cross-functional teams to deliver scalable data solutions, ensure data quality, and enable real-time and batch processing for large datasets in a Linux environment.