Spark & Kafka Big Data Platform Engineer

Quantum Technologies LLC; Chicago (IL); Vor Ort; USD 83.000 - 113.000; Vollzeit

Zusammenfassung

Quantum Technologies. LLC in Chicago is seeking an experienced Big Data Platform Engineer to design, administer, and optimize enterprise‑grade data platforms using Spark, HDFS, and Kafka.

Responsibilities include deploying, tuning, and supporting Spark, HDFS, and Kafka environments; ensuring security and compliance; and building automation with Python, Bash, or Perl. Strong Linux skills and collaboration with cross‑functional teams are essential for this 8‑month engagement.

Qualifikationen

• Minimum 3–5 years of production administration, configuration, performance tuning, and sizing experience with Spark, HDFS, and Kafka
• Solid Linux administration and command-line proficiency
• Experience with Grafana and Splunk for monitoring and observability
• Security foundations: Kerberos, SSL/TLS, IAM/LDAP integration
• Excellent analytical, problem-solving, and written/verbal communication skills
• Bachelor’s degree in Computer Science, Information Systems, or related field
• Experience with Cloudera Hadoop/CDP and Cloudera Manager preferred
• Cloudera certifications or equivalent big data expertise

Aufgaben

• Deploy, configure, monitor, tune, and troubleshoot Spark environments and workloads
• Manage HDFS storage including capacity, performance, and permissions
• Deploy, configure, monitor, and troubleshoot Kafka clusters and topics
• Monitor platform health and performance using Grafana, Splunk, and related tools
• Implement security configurations: Kerberos, SSL/TLS, IAM/LDAP integrations
• Develop automation scripts using Python, Bash, or Perl
• Provide on-call support and participate in platform lifecycle activities

Kenntnisse

Spark

Kafka

HDFS

Linux

Monitoring

Security basics

Communication

Ausbildung

Bachelor’s degree in Computer Science, Information Systems, or related field

Tools

Grafana

Splunk

Cloudera Manager

Python

Bash

Perl

Jobbeschreibung

Quantum Technologies. LLC in Chicago is seeking an experienced Big Data Platform Engineer to design, administer, and optimize enterprise‑grade data platforms using Spark, HDFS, and Kafka.

Responsibilities include deploying, tuning, and supporting Spark, HDFS, and Kafka environments; ensuring security and compliance; and building automation with Python, Bash, or Perl. Strong Linux skills and collaboration with cross‑functional teams are essential for this 8‑month engagement.