Big Data Platform Engineer

Valiant Technologies LLC

Chicago (IL)

On-site

USD 81,000 - 115,000

Full time

5 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Valiant Technologies LLC in Chicago is seeking a Big Data Platform Engineer to deploy and optimize Spark, HDFS, and Kafka in a production environment. You will work on security, monitoring, and automation to ensure reliable data platforms.

You will participate in on-call rotations, support platform lifecycle from installations to migrations, and collaborate with cross-functional teams to meet performance and compliance goals.

Qualifications

  • Minimum 3–5 years of hands-on production administration, configuration, performance tuning, and sizing experience with Apache Spark, HDFS, and Apache Kafka.
  • Solid Linux operating system administration and command-line proficiency.
  • Practical experience using Grafana and Splunk for monitoring and observability.
  • Solid understanding of Kerberos authentication, SSL/TLS encryption, IAM/LDAP integration, and security concepts for distributed data architectures.

Responsibilities

  • Deploy, configure, monitor, tune, and troubleshoot Spark environments and workloads.
  • Manage and support HDFS storage, capacity, performance, availability, permissions, replication, and troubleshooting.
  • Deploy, configure, manage, monitor, and troubleshoot Apache Kafka clusters.
  • Monitor platform connectivity, health, capacity, and performance using Grafana and Splunk.
  • Implement and manage security configurations including Kerberos, SSL/TLS, IAM/LDAP integrations.
  • Develop and maintain automation using Python, Bash, or Perl for routine administration and operations.
  • Provide expert-level troubleshooting and on-call support across the big data stack.
  • Support installations, patching, upgrades, migrations, and maintenance activities, including Cloudera Hadoop/CDP administration as needed.

Skills

Spark
HDFS
Kafka
Linux
Grafana
Splunk
Kerberos
IAM/LDAP integration
Communication

Education

Bachelor's degree in Computer Science / Information Systems
Cloudera certifications or equivalent

Tools

Cloudera Hadoop/CDP
Cloudera Manager

Job description

US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other Job Title:Big Data Platform Engineer
Location: Chicago,IL
Duration: 8 months
Bill Rate: $71/hour (C2C)
Job Type: W-2, 1099, C2C
Client: To Be Discussed Later
Work Authorization: US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, Other
Job Description:

Key Responsibilities:
  • Spark Administration: Deploy, configure, monitor, tune, and troubleshoot Apache Spark environments and workloads. Support Spark job performance, resource management, capacity planning, and production operations.
  • HDFS Administration: Manage and support HDFS storage, including capacity, performance, availability, permissions, replication, and troubleshooting. Ensure robust integration between distributed compute platforms and storage.
  • Kafka Administration: Deploy, configure, manage, monitor, and troubleshoot Apache Kafka clusters, topics, producers, and consumers. Support performance, capacity, availability, security, and production operations.
  • Monitoring & Performance Tuning: Monitor platform connectivity, health, capacity, and performance using tools such as Grafana, Splunk, and related management utilities. Proactively identify and resolve bottlenecks across Spark, HDFS, and Kafka.
  • Security & Compliance: Implement and manage security configurations, including Kerberos, SSL/TLS, authorization policies, HDFS ACLs, and enterprise IAM/LDAP integrations.
  • Automation & Scripting: Develop and maintain automation using Python, Bash, or Perl for routine administration, configuration management, monitoring, and operational support.
  • Troubleshooting & Support: Provide expert-level troubleshooting and technical support across the big data tech stack, including participation in an on-call rotation.
  • Platform Lifecycle: Support installations, patching, upgrades, migrations, and maintenance activities while minimizing downtime. Support Cloudera Hadoop/CDP administration as needed.
Qualifications & Requirements:
  • Core Experience: Minimum 3–5 years of hands-on production administration, configuration, performance tuning, and sizing experience with Apache Spark, HDFS, and Apache Kafka.
  • Linux Administration: Solid hands-on experience with Linux operating systems, logging in, navigating, and executing command-line administration.
  • Monitoring & Observability: Practical experience using enterprise monitoring and operational tooling such as Grafana and Splunk.
  • Security Foundations: Solid understanding of Kerberos authentication, SSL/TLS encryption, IAM/LDAP integration, and general security concepts for distributed data architectures.
  • Communication & Collaboration: Excellent analytical, problem-solving, and written/verbal communication skills.
  • Education: Bachelor’s degree in Computer Science, Information Systems, or a related field.
  • Experience with Cloudera Hadoop/CDP and Cloudera Manager (preferred, but secondary to foundational Spark, HDFS, and Kafka expertise).
  • Cloudera certifications or equivalent big data

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Big Data Platform Engineer
Big Data Platform Engineer

Robotics Technologies LLC • Chicago (IL)

On-site
USD 81,000 - 115,000
Big Data Platform Engineer
Big Data Platform Engineer

Ethereum Technologies LLC • Chicago (IL)

On-site
USD 81,000 - 115,000
Big Data Platform Engineer
Big Data Platform Engineer

Robotic Process Automation LLC • Chicago (IL)

On-site
USD 81,000 - 115,000
Big Data Platform Engineer
Big Data Platform Engineer

Quantum Technologies LLC • Chicago (IL)

On-site
USD 83,000 - 113,000
Bigdata Hadoop Administrator
Bigdata Hadoop Administrator

Ethereum Technologies LLC • Berkeley Heights (NJ)

On-site
USD 180,000 - 280,000
Bigdata Hadoop Administrator
Bigdata Hadoop Administrator

Valiant Technologies LLC • Berkeley Heights (NJ)

On-site
USD 150,000 - 230,000
Bigdata Hadoop Administrator
Bigdata Hadoop Administrator

Quantum Technologies LLC • New Jersey

On-site
USD 120,000 - 180,000
Big Data Platform Engineer
Big Data Platform Engineer

ZipStaff Inc. • Jersey City (NJ)

On-site
USD 120,000 - 180,000
Health Insurance
Dental Insurance
Vision Insurance
+3
Data Engineer
Data Engineer

Ethereum Technologies LLC • Dallas (TX)

On-site
USD 120,000 - 180,000
Big Data Engineer
Big Data Engineer

Robotics Technologies • Phoenix (AZ)

On-site
USD 83,000 - 118,000