Big Data Platform Engineer: Spark, Kafka, HDFS

Robotics Technologies LLC

Chicago (IL)

On-site

USD 81,000 - 115,000

Full time

4 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Robotics Technologies LLC in Chicago seeks a Big Data Platform Engineer to lead administration of Spark, HDFS, and Kafka ecosystems. The role emphasizes performance tuning, security, and automation in a multi-platform data environment.

You will manage monitoring, on-call support, and lifecycle activities, ensuring robust operations and scalable data infrastructure. A strong Linux background and formal CS-related education are required.

Qualifications

  • Minimum 3–5 years of hands-on production administration, configuration, performance tuning, and sizing with Spark, HDFS, and Kafka.
  • Solid Linux administration experience and command-line familiarity.
  • Experience with enterprise monitoring and observability tools (Grafana, Splunk).
  • Strong understanding of Kerberos, SSL/TLS, IAM/LDAP integrations, and security concepts for distributed data systems.
  • Excellent analytical, problem-solving, and written/verbal communication skills.
  • Bachelor’s degree in CS, IS, or related field.
  • Experience with Cloudera Hadoop/CDP and Cloudera Manager preferred, but secondary to core Spark/HDFS/Kafka expertise.
  • Cloudera certifications or equivalent big data knowledge.

Responsibilities

  • Deploy, configure, monitor, tune, and troubleshoot Spark environments and workloads for performance and operations.
  • Manage HDFS storage, capacity planning, permissions, replication, and integration with compute platforms.
  • Deploy, configure, monitor, and troubleshoot Kafka clusters, topics, producers, and consumers.
  • Monitor platform health and performance; identify bottlenecks across Spark, HDFS, and Kafka.
  • Implement security configurations including Kerberos, SSL/TLS, and IAM/LDAP integrations.
  • Develop automation using Python, Bash, or Perl for routine admin tasks.
  • Provide expert-level troubleshooting and on-call support across the data stack.
  • Support installations, patching, upgrading, migrations, and maintenance activities.

Skills

Spark
HDFS
Kafka
Monitoring
Security
Automation
Troubleshooting
Platform Lifecycle

Education

Bachelor’s degree

Tools

Grafana
Splunk

Job description

Robotics Technologies LLC in Chicago seeks a Big Data Platform Engineer to lead administration of Spark, HDFS, and Kafka ecosystems. The role emphasizes performance tuning, security, and automation in a multi-platform data environment.

You will manage monitoring, on-call support, and lifecycle activities, ensuring robust operations and scalable data infrastructure. A strong Linux background and formal CS-related education are required.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Big Data Platform Engineer — Spark, HDFS & Kafka Ops
Big Data Platform Engineer — Spark, HDFS & Kafka Ops

Robotic Process Automation LLC • Chicago (IL)

On-site
USD 81,000 - 115,000
Big Data Platform Engineer (Spark/HDFS/Kafka) – Chicago
Big Data Platform Engineer (Spark/HDFS/Kafka) – Chicago

Ethereum Technologies LLC • Chicago (IL)

On-site
USD 81,000 - 115,000
Spark & Kafka Big Data Platform Engineer
Spark & Kafka Big Data Platform Engineer

Quantum Technologies LLC • Chicago (IL)

On-site
USD 83,000 - 113,000
Chicago Big Data Platform Engineer (Spark, Kafka)
Chicago Big Data Platform Engineer (Spark, Kafka)

Valiant Technologies LLC • Chicago (IL)

On-site
USD 81,000 - 115,000
Big Data Platform Engineer
Big Data Platform Engineer

Valiant Technologies LLC • Chicago (IL)

On-site
USD 81,000 - 115,000
Big Data Platform Engineer
Big Data Platform Engineer

Robotics Technologies LLC • Chicago (IL)

On-site
USD 81,000 - 115,000
Big Data Platform Engineer
Big Data Platform Engineer

Ethereum Technologies LLC • Chicago (IL)

On-site
USD 81,000 - 115,000
Senior Hadoop & Spark Data Engineer (Cloud + Kafka)
Senior Hadoop & Spark Data Engineer (Cloud + Kafka)

Robotic Technologies Automation • Arizona

On-site
USD 120,000 - 160,000
Big Data Platform Engineer
Big Data Platform Engineer

Quantum Technologies LLC • Chicago (IL)

On-site
USD 83,000 - 113,000
Big Data Platform Engineer
Big Data Platform Engineer

Robotic Process Automation LLC • Chicago (IL)

On-site
USD 81,000 - 115,000