Big Data Platform Engineer

Ethereum Technologies; Chicago (IL); On-site; USD 81,000 - 115,000; Part time

Job summary

Ethereum Technologies is seeking a Big Data Platform Engineer in Chicago to administer Spark, HDFS, and Kafka environments. You will monitor, tune, and secure a distributed data platform, integrating Kerberos, SSL/TLS, and IAM/LDAP.

The role requires strong scripting skills and experience with Grafana/Splunk for performance visibility. Responsibilities include platform lifecycle support, on-call rotation, and collaboration across teams to ensure reliable data processing, storage, and analytics

Qualifications

• Minimum 3–5 years of hands-on production administration with Spark, HDFS, and Kafka.
• Solid Linux administration experience and command-line proficiency.
• Experience with monitoring/observability tools (Grafana, Splunk).
• Strong understanding of Kerberos, SSL/TLS, IAM/LDAP integration and security concepts.
• Excellent communication and collaboration skills.
• Bachelor’s degree in Computer Science, Information Systems, or related field.

Responsibilities

• Deploy, configure, monitor, tune, and troubleshoot Spark environments and workloads.
• Manage HDFS storage, capacity, performance, and replication.
• Deploy, configure, monitor, and troubleshoot Kafka clusters, topics, producers, and consumers.
• Monitor and optimize platform performance using Grafana, Splunk, and related tools.
• Implement security configurations including Kerberos and IAM/LDAP integrations.
• Develop automation with Python, Bash, or Perl for admin tasks.
• Provide on-call support and troubleshoot end-to-end big data stack.
• Support Cloudera Hadoop/CDP installations, patching, upgrades, and migrations.

Skills

Spark

HDFS

Kafka

Linux

Kerberos

SSL/TLS

IAM/LDAP

Grafana

Splunk

Education

Bachelor’s degree in Computer Science, Information Systems, or a related field

Tools

Cloudera Hadoop/CDP

Cloudera Manager

Grafana

Splunk

Job description

US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other Job Title:Big Data Platform Engineer
Location: Chicago,IL
Duration: 8 months
Bill Rate: $71/hour (C2C)
Job Type: W-2, 1099, C2C
Client: To Be Discussed Later
Work Authorization: US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, Other
Job Description:

Key Responsibilities:

• Spark Administration: Deploy, configure, monitor, tune, and troubleshoot Apache Spark environments and workloads. Support Spark job performance, resource management, capacity planning, and production operations.
• HDFS Administration: Manage and support HDFS storage, including capacity, performance, availability, permissions, replication, and troubleshooting. Ensure robust integration between distributed compute platforms and storage.
• Kafka Administration: Deploy, configure, manage, monitor, and troubleshoot Apache Kafka clusters, topics, producers, and consumers. Support performance, capacity, availability, security, and production operations.
• Monitoring & Performance Tuning: Monitor platform connectivity, health, capacity, and performance using tools such as Grafana, Splunk, and related management utilities. Proactively identify and resolve bottlenecks across Spark, HDFS, and Kafka.
• Security & Compliance: Implement and manage security configurations, including Kerberos, SSL/TLS, authorization policies, HDFS ACLs, and enterprise IAM/LDAP integrations.
• Automation & Scripting: Develop and maintain automation using Python, Bash, or Perl for routine administration, configuration management, monitoring, and operational support.
• Troubleshooting & Support: Provide expert-level troubleshooting and technical support across the big data tech stack, including participation in an on-call rotation.
• Platform Lifecycle: Support installations, patching, upgrades, migrations, and maintenance activities while minimizing downtime. Support Cloudera Hadoop/CDP administration as needed.

Qualifications & Requirements:

• Core Experience: Minimum 3–5 years of hands-on production administration, configuration, performance tuning, and sizing experience with Apache Spark, HDFS, and Apache Kafka.
• Linux Administration: Solid hands-on experience with Linux operating systems, logging in, navigating, and executing command-line administration.
• Monitoring & Observability: Practical experience using enterprise monitoring and operational tooling such as Grafana and Splunk.
• Security Foundations: Solid understanding of Kerberos authentication, SSL/TLS encryption, IAM/LDAP integration, and general security concepts for distributed data architectures.
• Communication & Collaboration: Excellent analytical, problem-solving, and written/verbal communication skills.
• Education: Bachelor’s degree in Computer Science, Information Systems, or a related field.
• Experience with Cloudera Hadoop/CDP and Cloudera Manager (preferred, but secondary to foundational Spark, HDFS, and Kafka expertise).
• Cloudera certifications or equivalent big data

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.