Big Data Developer

VeriiPro; Jersey City (NJ); On-site; USD 120,000 - 180,000; Full time

Job summary

VeriiPro is seeking an experienced Big Data Developer to design, build, and optimize data processing pipelines using Kafka, Spark, Scala, SQL, Hadoop, Hive, and Impala. The role focuses on performance tuning and troubleshooting within the Cloudera Big Data ecosystem.

You will work with cross-functional teams to deliver scalable data solutions, ensure data quality, and enable real-time and batch processing for large datasets in a Linux environment.

Qualifications

• Experience with large-scale data processing and distributed systems.
• Hands-on work with Kafka, Spark, SQL, Hive and Impala in Cloudera ecosystems.
• Strong troubleshooting and performance tuning skills for big data jobs.
• Proficiency in Linux-based environments and batch/workflow scheduling.

Responsibilities

• Develop, maintain, and optimize big data applications and data processing pipelines.
• Work with Kafka and Apache Spark for real-time and batch data processing.
• Perform data analysis, data wrangling, and processing of large volumes of structured and unstructured data.
• Write and optimize complex SQL queries using Hive and Impala.
• Monitor, troubleshoot, and resolve performance issues in the Cloudera Big Data ecosystem.
• Use Splunk for log analysis, monitoring, and troubleshooting.
• Manage and monitor batch workflows and job scheduling using AutoSys.
• Work in Linux-based environments for development, deployment, and operational support.
• Collaborate with business users and cross-functional teams to deliver high-quality data solutions.
• Ensure data quality, reliability, scalability, and performance of big data applications.

Skills

Kafka

Spark

Scala

SQL

Data analysis

Data wrangling

Performance tuning

Troubleshooting

Linux

Distributed data processing

Tools

Hadoop

Hive

Impala

Cloudera

Splunk

AutoSys

Job description

We are looking for an experienced Big Data Developer with strong expertise in big data technologies, distributed data processing, and large-scale data analysis. The ideal candidate should have hands-on experience with Kafka, Spark, Scala, SQL, Hadoop, Hive, and Impala, along with performance tuning and troubleshooting in the Cloudera Big Data ecosystem.

Key Responsibilities

We are looking for an experienced Big Data Developer with strong expertise in big data technologies, distributed data processing, and large-scale data analysis. The ideal candidate should have hands-on experience with Kafka, Spark, Scala, SQL, Hadoop, Hive, and Impala, along with performance tuning and troubleshooting in the Cloudera Big Data ecosystem.

Key Responsibilities

• Develop, maintain, and optimize big data applications and data processing pipelines.
• Work with Kafka and Apache Spark for real-time and batch data processing.
• Perform data analysis, data wrangling, and processing of large volumes of structured and unstructured data.
• Write and optimize complex SQL queries using Hive and Impala.
• Monitor, troubleshoot, and resolve performance issues in the Cloudera Big Data ecosystem.
• Use Splunk for log analysis, monitoring, and troubleshooting.
• Manage and monitor batch workflows and job scheduling using AutoSys.
• Work in Linux-based environments for development, deployment, and operational support.
• Collaborate with business users and cross-functional teams to deliver high-quality data solutions.
• Ensure data quality, reliability, scalability, and performance of big data applications.

Required Technical Skills

• Strong hands-on experience with Kafka, Spark, Scala, and SQL.
• Experience with Hive, Impala, Hadoop, and the Cloudera ecosystem.
• Proficiency in data analysis and data wrangling for large datasets.
• Experience in performance analysis, tuning, troubleshooting, and issue resolution.
• Hands-on experience with Splunk and AutoSys.
• Strong knowledge of Linux-based infrastructures.
• Experience in distributed data processing and big data technologies.