Senior Big Data Engineer — Onsite Chicago, Spark & Kafka

Select Minds LLC; Lansing (MI); On-site; USD 100,000 - 160,000; Full time

Benefits offered by this job

Onsite

Competitive salary

Opportunity for advancement

Job summary

Select Minds LLC is seeking an experienced Big Data Engineer in Chicago, IL (Day 1 onsite) to design, develop, and optimize enterprise-scale data solutions. You will work with Spark, Scala, Hive, NiFi, and Kafka to build scalable pipelines and real‑time streaming while ensuring data quality and governance.

The role requires 8+ years in Big Data, client-facing experience, and strong CI/CD practices. You'll collaborate with Agile teams, implement CI/CD with Git/Jenkins/Nexus, and mentor junior

Qualifications

• 8+ years of experience in Big Data technologies.
• Strong hands-on experience with Spark, Scala, Hive and the Hadoop ecosystem.
• Experience implementing Apache NiFi and Apache Kafka.
• Proficiency in Shell scripting or Python.
• Experience with PostgreSQL.
• Familiarity with scheduling tools such as Oozie and Control-M.
• Working knowledge of Git, Jenkins, Nexus, Maven, Ansible, and SonarQube.
• Strong CI/CD practices.
• Experience in client-facing roles, including requirement gathering workshops.
• Minimum of two end-to-end Big Data implementation projects.

Responsibilities

• Design, develop, and optimize large-scale data processing pipelines using Spark, Scala, and Hive.
• Build and maintain data ingestion workflows using Apache NiFi.
• Develop and support real‑time data streaming solutions using Apache Kafka.
• Monitor application logs and system health using Kibana, Elasticsearch, and Logstash.
• Implement and support CI/CD pipelines using Git, Jenkins, and Nexus.
• Ensure data quality, integrity, and governance across Big Data platforms.
• Participate in architecture discussions and recommend scalable design solutions.
• Troubleshoot and resolve complex production issues across the Big Data stack.
• Collaborate with cross-functional teams in an Agile environment.
• Mentor junior engineers and contribute to technical best practices.

Skills

Spark

Scala

Hive

Hadoop

NiFi

Kafka

Python

Shell scripting

PostgreSQL

Oozie

Control-M

Git

Jenkins

Nexus

Maven

Ansible

SonarQube

CI/CD

Tools

Git

Jenkins

Nexus

Maven

Ansible

SonarQube

Job description

Select Minds LLC is seeking an experienced Big Data Engineer in Chicago, IL (Day 1 onsite) to design, develop, and optimize enterprise-scale data solutions. You will work with Spark, Scala, Hive, NiFi, and Kafka to build scalable pipelines and real‑time streaming while ensuring data quality and governance.

The role requires 8+ years in Big Data, client-facing experience, and strong CI/CD practices. You'll collaborate with Agile teams, implement CI/CD with Git/Jenkins/Nexus, and mentor junior