Chicago Big Data Platform Engineer (Spark, Kafka)

Valiant Technologies LLC

Chicago (IL)

On-site

USD 81,000 - 115,000

Full time

5 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Valiant Technologies LLC in Chicago is seeking a Big Data Platform Engineer to deploy and optimize Spark, HDFS, and Kafka in a production environment. You will work on security, monitoring, and automation to ensure reliable data platforms.

You will participate in on-call rotations, support platform lifecycle from installations to migrations, and collaborate with cross-functional teams to meet performance and compliance goals.

Qualifications

  • Minimum 3–5 years of hands-on production administration, configuration, performance tuning, and sizing experience with Apache Spark, HDFS, and Apache Kafka.
  • Solid Linux operating system administration and command-line proficiency.
  • Practical experience using Grafana and Splunk for monitoring and observability.
  • Solid understanding of Kerberos authentication, SSL/TLS encryption, IAM/LDAP integration, and security concepts for distributed data architectures.

Responsibilities

  • Deploy, configure, monitor, tune, and troubleshoot Spark environments and workloads.
  • Manage and support HDFS storage, capacity, performance, availability, permissions, replication, and troubleshooting.
  • Deploy, configure, manage, monitor, and troubleshoot Apache Kafka clusters.
  • Monitor platform connectivity, health, capacity, and performance using Grafana and Splunk.
  • Implement and manage security configurations including Kerberos, SSL/TLS, IAM/LDAP integrations.
  • Develop and maintain automation using Python, Bash, or Perl for routine administration and operations.
  • Provide expert-level troubleshooting and on-call support across the big data stack.
  • Support installations, patching, upgrades, migrations, and maintenance activities, including Cloudera Hadoop/CDP administration as needed.

Skills

Spark
HDFS
Kafka
Linux
Grafana
Splunk
Kerberos
IAM/LDAP integration
Communication

Education

Bachelor's degree in Computer Science / Information Systems
Cloudera certifications or equivalent

Tools

Cloudera Hadoop/CDP
Cloudera Manager

Job description

Valiant Technologies LLC in Chicago is seeking a Big Data Platform Engineer to deploy and optimize Spark, HDFS, and Kafka in a production environment. You will work on security, monitoring, and automation to ensure reliable data platforms.

You will participate in on-call rotations, support platform lifecycle from installations to migrations, and collaborate with cross-functional teams to meet performance and compliance goals.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Spark & Kafka Big Data Platform Engineer
Spark & Kafka Big Data Platform Engineer

Quantum Technologies LLC • Chicago (IL)

On-site
USD 83,000 - 113,000
Big Data Platform Engineer (Spark/HDFS/Kafka) – Chicago
Big Data Platform Engineer (Spark/HDFS/Kafka) – Chicago

Ethereum Technologies LLC • Chicago (IL)

On-site
USD 81,000 - 115,000
Big Data Platform Engineer: Spark, Kafka, HDFS
Big Data Platform Engineer: Spark, Kafka, HDFS

Robotics Technologies LLC • Chicago (IL)

On-site
USD 81,000 - 115,000
Big Data Platform Engineer — Spark, HDFS & Kafka Ops
Big Data Platform Engineer — Spark, HDFS & Kafka Ops

Robotic Process Automation LLC • Chicago (IL)

On-site
USD 81,000 - 115,000
Senior Big Data Spark & Hadoop Administrator
Senior Big Data Spark & Hadoop Administrator

Valiant Technologies LLC • Berkeley Heights (NJ)

On-site
USD 150,000 - 230,000
Hadoop & Spark Engineer - Cloud Data Platform
Hadoop & Spark Engineer - Cloud Data Platform

Valiant Technologies LLC • Bentonville (AR)

On-site
USD 110,000 - 150,000
Data Engineer - Build Scalable Pipelines in Chicago
Data Engineer - Build Scalable Pipelines in Chicago

Valiant Technologies LLC • Chicago (IL)

On-site
USD 90,000 - 110,000
Senior Data Engineer - Cloud Data Platform (AWS, Spark)
Senior Data Engineer - Cloud Data Platform (AWS, Spark)

Valiant Technologies LLC • Harrisburg

On-site
USD 95,000 - 150,000
Big Data Platform Engineer
Big Data Platform Engineer

Valiant Technologies LLC • Chicago (IL)

On-site
USD 81,000 - 115,000
Big Data Platform Engineer
Big Data Platform Engineer

Robotics Technologies LLC • Chicago (IL)

On-site
USD 81,000 - 115,000