Big Data Engineer

Select Minds LLC; Lansing (MI); On-site; USD 100,000 - 160,000; Full time

Benefits offered by this job

Onsite

Competitive salary

Opportunity for advancement

Job summary

Select Minds LLC is seeking an experienced Big Data Engineer in Chicago, IL (Day 1 onsite) to design, develop, and optimize enterprise-scale data solutions. You will work with Spark, Scala, Hive, NiFi, and Kafka to build scalable pipelines and real‑time streaming while ensuring data quality and governance.

The role requires 8+ years in Big Data, client-facing experience, and strong CI/CD practices. You'll collaborate with Agile teams, implement CI/CD with Git/Jenkins/Nexus, and mentor junior

Qualifications

• 8+ years of experience in Big Data technologies.
• Strong hands-on experience with Spark, Scala, Hive and the Hadoop ecosystem.
• Experience implementing Apache NiFi and Apache Kafka.
• Proficiency in Shell scripting or Python.
• Experience with PostgreSQL.
• Familiarity with scheduling tools such as Oozie and Control-M.
• Working knowledge of Git, Jenkins, Nexus, Maven, Ansible, and SonarQube.
• Strong CI/CD practices.
• Experience in client-facing roles, including requirement gathering workshops.
• Minimum of two end-to-end Big Data implementation projects.

Responsibilities

• Design, develop, and optimize large-scale data processing pipelines using Spark, Scala, and Hive.
• Build and maintain data ingestion workflows using Apache NiFi.
• Develop and support real‑time data streaming solutions using Apache Kafka.
• Monitor application logs and system health using Kibana, Elasticsearch, and Logstash.
• Implement and support CI/CD pipelines using Git, Jenkins, and Nexus.
• Ensure data quality, integrity, and governance across Big Data platforms.
• Participate in architecture discussions and recommend scalable design solutions.
• Troubleshoot and resolve complex production issues across the Big Data stack.
• Collaborate with cross-functional teams in an Agile environment.
• Mentor junior engineers and contribute to technical best practices.

Skills

Spark

Scala

Hive

Hadoop

NiFi

Kafka

Python

Shell scripting

PostgreSQL

Oozie

Control-M

Git

Jenkins

Nexus

Maven

Ansible

SonarQube

CI/CD

Tools

Git

Jenkins

Nexus

Maven

Ansible

SonarQube

Job description

Benefits:

• ONSITE
• Competitive salary
• Opportunity for advancement

Big Data Engineer Location: Chicago, IL (Day 1 Onsite) Job Type: Contract / Full Time Experience: 8+ yrs Duration: Long Term

We are seeking an experienced Big Data Engineer to design, develop, and optimize enterprise-scale big data solutions. The ideal candidate will have strong hands‑on expertise in the Hadoop ecosystem, Spark, Scala, Hive, Apache NiFi, and Kafka, along with experience building scalable data pipelines and supporting modern CI/CD practices. Candidates should have worked directly with clients, participated in requirement gathering sessions, and delivered end‑to‑end Big Data implementations.

Key Responsibilities

• Design, develop, and optimize large-scale data processing pipelines using Spark, Scala, and Hive.
• Build and maintain data ingestion workflows using Apache NiFi.
• Develop and support real‑time data streaming solutions using Apache Kafka.
• Monitor application logs and system health using Kibana, Elasticsearch, and Logstash.
• Implement and support CI/CD pipelines using Git, Jenkins, and Nexus.
• Ensure data quality, integrity, and governance across Big Data platforms.
• Participate in architecture discussions and recommend scalable design solutions.
• Troubleshoot and resolve complex production issues across the Big Data stack.
• Collaborate with cross-functional teams in an Agile environment.
• Mentor junior engineers and contribute to technical best practices.

Required Qualifications

• 8+ years of experience in Big Data technologies.
• Strong hands‑on experience with Spark, Scala, Hive (HQL), and the Hadoop ecosystem.
• Experience implementing Apache NiFi and Apache Kafka.
• Proficiency in Shell Scripting or Python.
• Experience with PostgreSQL.
• Familiarity with scheduling tools such as Oozie and Control-M.
• Working knowledge of Git, Jenkins, Nexus, Maven, Ansible, and SonarQube.
• Strong understanding of CI/CD practices.
• Experience in client‑facing roles, including requirement gathering workshops.
• Minimum of two end‑to‑end Big Data implementation projects.
• Ability to work independently and contribute effectively within Agile teams.