Big Data Platform Engineer: Spark, HDFS & Kafka (Onsite)

ZipStaff Inc.; Jersey City (NJ); On-site; USD 120,000 - 180,000; Full time

Benefits offered by this job

Health Insurance

Dental Insurance

Vision Insurance

401(k) + matching

Paid time off

Referral Program

Job summary

ZipStaff is hiring a Big Data Platform Engineer for our client, a leading global financial services firm, for a 12-month onsite contract in Jersey City. You will manage, support, and optimize enterprise-grade big data platforms with hands-on expertise in Spark, HDFS, and Kafka, administering production environments in a Cloudera context.

The role emphasizes monitoring, performance tuning, security, and lifecycle operations, plus on-call rotation and automation with Python/Bash/Perl.

Qualifications

• 3+ years of hands-on production administration experience with Spark, HDFS, and Kafka.
• Strong demonstrable expertise in Spark administration, tuning, and troubleshooting.
• Strong demonstrable expertise in HDFS administration, including storage management, security, and performance.
• Strong demonstrable expertise in Kafka administration, including topic management, producer/consumer architecture, performance, and security.
• Strong Linux and command-line skills.
• Experience with Grafana and Splunk for monitoring and performance analysis.
• Solid knowledge of Kerberos, SSL/TLS, and enterprise IAM/LDAP.
• Excellent problem-solving, analytical, and communication skills.

Responsibilities

• Deploy, configure, monitor, tune, and troubleshoot Apache Spark environments and workloads, including job performance, resource management, capacity planning, and production operations.
• Administer HDFS for capacity, performance, availability, permissions, replication, and troubleshooting, with strong understanding of HDFS architecture and integration with distributed compute platforms.
• Deploy, configure, manage, monitor, and troubleshoot Kafka clusters, topics, producers, and consumers, covering performance, capacity, availability, security, and production operations.
• Monitor and tune platform performance using Grafana, Splunk, and related tools, and resolve bottlenecks across Spark, HDFS, and Kafka.
• Implement and maintain security and compliance controls, including Kerberos, SSL/TLS, authorization policies, HDFS ACLs, and enterprise IAM/LDAP integration.
• Automate operational tasks using Python, Bash, or Perl.
• Provide expert troubleshooting and production support, including participation in an on‑call rotation.
• Perform platform administration including installs, patching, upgrades, migrations, and lifecycle management with minimal downtime; Cloudera Hadoop/CDP administration is a secondary focus.

Skills

Spark

HDFS

Kafka

Linux

Grafana

Splunk

Kerberos

SSL/TLS

IAM/LDAP

Python

Bash

Perl

On-call

Tools

Cloudera Hadoop/CDP

Cloudera Manager

Job description

ZipStaff is hiring a Big Data Platform Engineer for our client, a leading global financial services firm, for a 12-month onsite contract in Jersey City. You will manage, support, and optimize enterprise-grade big data platforms with hands-on expertise in Spark, HDFS, and Kafka, administering production environments in a Cloudera context.

The role emphasizes monitoring, performance tuning, security, and lifecycle operations, plus on-call rotation and automation with Python/Bash/Perl.