Big Data Platform Engineer

ZipStaff Inc.; Jersey City (NJ); On-site; USD 120,000 - 180,000; Full time

Benefits offered by this job

Health Insurance

Dental Insurance

Vision Insurance

401(k) + matching

Paid time off

Referral Program

Job summary

ZipStaff is hiring a Big Data Platform Engineer for our client, a leading global financial services firm, for a 12-month onsite contract in Jersey City. You will manage, support, and optimize enterprise-grade big data platforms with hands-on expertise in Spark, HDFS, and Kafka, administering production environments in a Cloudera context.

The role emphasizes monitoring, performance tuning, security, and lifecycle operations, plus on-call rotation and automation with Python/Bash/Perl.

Qualifications

• 3+ years of hands-on production administration experience with Spark, HDFS, and Kafka.
• Strong demonstrable expertise in Spark administration, tuning, and troubleshooting.
• Strong demonstrable expertise in HDFS administration, including storage management, security, and performance.
• Strong demonstrable expertise in Kafka administration, including topic management, producer/consumer architecture, performance, and security.
• Strong Linux and command-line skills.
• Experience with Grafana and Splunk for monitoring and performance analysis.
• Solid knowledge of Kerberos, SSL/TLS, and enterprise IAM/LDAP.
• Excellent problem-solving, analytical, and communication skills.

Responsibilities

• Deploy, configure, monitor, tune, and troubleshoot Apache Spark environments and workloads, including job performance, resource management, capacity planning, and production operations.
• Administer HDFS for capacity, performance, availability, permissions, replication, and troubleshooting, with strong understanding of HDFS architecture and integration with distributed compute platforms.
• Deploy, configure, manage, monitor, and troubleshoot Kafka clusters, topics, producers, and consumers, covering performance, capacity, availability, security, and production operations.
• Monitor and tune platform performance using Grafana, Splunk, and related tools, and resolve bottlenecks across Spark, HDFS, and Kafka.
• Implement and maintain security and compliance controls, including Kerberos, SSL/TLS, authorization policies, HDFS ACLs, and enterprise IAM/LDAP integration.
• Automate operational tasks using Python, Bash, or Perl.
• Provide expert troubleshooting and production support, including participation in an on‑call rotation.
• Perform platform administration including installs, patching, upgrades, migrations, and lifecycle management with minimal downtime; Cloudera Hadoop/CDP administration is a secondary focus.

Skills

Spark

HDFS

Kafka

Linux

Grafana

Splunk

Kerberos

SSL/TLS

IAM/LDAP

Python

Bash

Perl

On-call

Tools

Cloudera Hadoop/CDP

Cloudera Manager

Job description

Title:

Big Data Platform Engineer.

Location:

Jersey City, NJ (Onsite).

Type:

Contract, 12 months.

Overview:

ZipStaff is hiring a Big Data Platform Engineer for our client, a leading global financial services firm. This 12-month contract role focuses on managing, supporting, and optimizing enterprise-grade big data platforms, with deep hands‑on expertise in Apache Spark, HDFS, and Kafka as the primary focus. You will administer production Spark environments, HDFS clusters, and Kafka infrastructure, including monitoring, performance tuning, security, and lifecycle operations. Experience administering these technologies within a Cloudera environment is preferred but secondary to strong expertise in the underlying platforms. Candidates must be local to the Jersey City, NJ area; this is a fully onsite role.

Responsibilities:

• Deploy, configure, monitor, tune, and troubleshoot Apache Spark environments and workloads, including job performance, resource management, capacity planning, and production operations.
• Administer HDFS for capacity, performance, availability, permissions, replication, and troubleshooting, with strong understanding of HDFS architecture and integration with distributed compute platforms.
• Deploy, configure, manage, monitor, and troubleshoot Kafka clusters, topics, producers, and consumers, covering performance, capacity, availability, security, and production operations.
• Monitor and tune platform performance using Grafana, Splunk, and related tools, and resolve bottlenecks across Spark, HDFS, and Kafka.
• Implement and maintain security and compliance controls, including Kerberos, SSL/TLS, authorization policies, HDFS ACLs, and enterprise IAM/LDAP integration.
• Automate operational tasks using Python, Bash, or Perl.
• Provide expert troubleshooting and production support, including participation in an on‑call rotation.
• Perform platform administration including installs, patching, upgrades, migrations, and lifecycle management with minimal downtime; Cloudera Hadoop/CDP administration is a secondary focus.

Required Qualifications:

• 3+ years of hands‑on production administration experience with Apache Spark, HDFS, and Kafka.
• Strong demonstrable expertise in Spark administration, tuning, and troubleshooting.
• Strong demonstrable expertise in HDFS administration, including storage management, security, and performance.
• Strong demonstrable expertise in Kafka administration, including topic management, producer/consumer architecture, performance, and security.
• Strong Linux and command‑line skills.
• Experience with Grafana and Splunk for monitoring and performance analysis.
• Solid knowledge of Kerberos, SSL/TLS, and enterprise IAM/LDAP.
• Excellent problem‑solving, analytical, and communication skills.

Preferred Qualifications:

• Experience administering Spark, HDFS, and Kafka within a Cloudera Hadoop/CDP environment, including Cloudera Manager.
• Cloudera or equivalent certifications.
• Automation experience with Python, Bash, or Perl.

Benefits:

• Health Insurance.
• Dental Insurance.
• Vision Insurance.
• 401(k) + matching.
• Paid time off.
• Referral Program.

About ZipStaff:

ZipStaff connects skilled technology professionals with leading organizations across the United States, matching top talent with impactful contract and full‑time opportunities.