Spark, HDFS & Kafka Administrator/Engineer - NJ onsite

Yoh; Jersey City (NJ); Hybrid; USD 68,000 - 96,000; Full time

Benefits offered by this job

Medical Benefits

401K Retirement Savings Plan

Life & Disability Insurance

Employee Assistance Program (EAP)

Direct Deposit & weekly epayroll

Certification and training

Job summary

Yoh is seeking an experienced Spark, HDFS, and Kafka Administrator/Engineer to manage and optimize enterprise-grade big data platforms. The role focuses on Spark, HDFS, and Kafka within a Cloudera environment, with broader platform duties as needed.

The candidate will deploy, configure, monitor, and tune Spark, HDFS, and Kafka workloads, ensure security and compliance, and participate in on-call support. Hybrid onsite model in Jersey City with competitive benefits.

Qualifications

• Excellent problem-solving, analytical, and communication skills.
• Hybrid onsite 4 days a week.
• Minimum 3 years of hands-on production experience administering and supporting Apache Spark, HDFS, and Kafka.
• Strong, demonstrable expertise in Spark

Responsibilities

• Spark Administration: Deploy, configure, monitor, tune, and troubleshoot Spark environments and workloads.
• HDFS Administration: Manage storage, capacity, performance, permissions, replication, and troubleshooting.
• Kafka Administration: Deploy, configure, manage, monitor, and troubleshoot Kafka clusters, topics, producers, and consumers.
• Monitoring and Performance Tuning: Monitor connectivity, health, capacity, and performance using Grafana and Splunk.
• Automation and Scripting: Develop automation using Python, Bash, or Perl for routine tasks.

Skills

Communication

Problem-solving

Analytical thinking

Hands-on

Tools

Grafana

Splunk

Python

Bash

Perl

Job description

Onsite in Jersey City

Must Haves:

• Great Communication, able to explain everything in detail
• Linux Admin - log in and perform commands
• Hands-on 3-5 years min experience Cloudera Cluster, HDFS, Spark, Kafka

We are seeking an experienced Spark, HDFS, and Kafka Administrator/Engineer to manage, support, and optimize enterprise-grade big data platforms. Deep hands-on expertise with Apache Spark, HDFS, and Kafka is required and will be the primary focus of this role. Experience administering these technologies within a Cloudera environment is preferred but secondary to strong expertise in the underlying technologies.

Main areas of responsibilities:

• Spark Administration: Deploy, configure, monitor, tune, and troubleshoot Apache Spark environments and workloads. Support Spark job performance, resource management, capacity planning, and production operations.
• HDFS Administration: Manage and support HDFS storage, including capacity, performance, availability, permissions, replication, and troubleshooting. Strong understanding of HDFS architecture and integration with distributed compute platforms is required.
• Kafka Administration: Deploy, configure, manage, monitor, and troubleshoot Apache Kafka clusters, topics, producers, and consumers. Support performance, capacity, availability, security, and production operations.
• Monitoring and Performance Tuning: Monitor platform connectivity, health, capacity, and performance using tools such as Grafana, Splunk, and relevant platform management tools. Proactively identify and resolve performance bottlenecks across Spark, HDFS, and Kafka.
• Security and Compliance: Implement and manage security configurations, including Kerberos, SSL/TLS, authorization policies, HDFS ACLs, and enterprise IAM/LDAP integrations.
• Automation and Scripting: Develop and maintain automation using Python, Bash, or Perl for routine administration, configuration management, monitoring, and operational support.
• Troubleshooting and Support: Provide expert-level troubleshooting and technical support for Spark, HDFS, Kafka, and related big data technologies, including participation in an on-call rotation.
• Platform Administration: Support installations, patching, upgrades, migrations, and platform lifecycle activities while minimizing downtime. Cloudera Hadoop/CDP administration will be part of the role but is secondary to expertise in Spark, HDFS, and Kafka.

Requirements:

• Excellent problem-solving, analytical, and communication skills.
• Hybrid onsite 4 days a week.
• Minimum 3 years of hands-on production experience administering and supporting Apache Spark, HDFS, and Kafka.
• Strong, demonstrable expertise in Spark

Estimated Min Rate: $49.00

Estimated Max Rate: $70.00

What’s In It for You?

We welcome you to be a part of the largest and legendary global staffing companies to meet your career aspirations. Yoh’s network of client companies has been employing professionals like you for over 65 years in the U.S., UK and Canada. Join Yoh’s extensive talent community that will provide you with access to Yoh’s vast network of opportunities and gain access to this exclusive opportunity available to you. Benefit eligibility is in accordance with applicable laws and client requirements. Benefits include:

• Medical, Prescription, Dental & Vision Benefits (for employees working 20+ hours per week)
• Health Savings Account (HSA) (for employees working 20+ hours per week)
• Life & Disability Insurance (for employees working 20+ hours per week)
• MetLife Voluntary Benefits
• Employee Assistance Program (EAP)
• 401K Retirement Savings Plan
• Direct Deposit & weekly epayroll
• Referral Bonus Programs
• Certification and training opportunities

Note: Any pay ranges displayed are estimations. Actual pay is determined by an applicant's experience, technical expertise, and other qualifications as listed in the job description.

Yoh, a Day & Zimmermann company, is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.

Visit https://www.yoh.com/applicants-with-disabilities to contact us if you are an individual with a disability and require accommodation in the application process.

For California applicants, qualified applicants with arrest or conviction records will be considered for employment in accordance with the Los Angeles County Fair Chance Ordinance for Employers and the California Fair Chance Act. All of the material job duties described in this posting are job duties for which a criminal history may have a direct, adverse, and negative relationship potentially resulting in the withdrawal of a conditional offer of employment.

It is unlawful in Massachusetts to require or administer a lie detector test as a condition of employment or continued employment. An employer who violates this law shall be subject to criminal penalties and civil liability.