Senior Big Data Engineer - Spark, Scala & Streaming

Infinite Computer Solutions; Town of Texas (WI); On-site; USD 120,000 - 170,000; Full time

Job summary

Infinite Computer Solutions is seeking an experienced Big Data Engineer to design, build, and optimize large-scale batch and streaming data pipelines on the Hadoop ecosystem using Apache Spark and Scala.

You will work with data architects and analytics teams to ingest, transform, and enrich clickstream, network, and location datasets, ensuring data quality and reliability in production environments.

Qualifications

• 6+ years of data engineering experience with 4+ years Spark in Scala on production workloads.
• Strong Scala fundamentals and functional programming concepts.
• Deep Hadoop ecosystem knowledge: HDFS, Hive, YARN, HBase.
• Advanced SQL and data modeling skills for big data denormalized patterns.
• Experience with Spark performance tuning and debugging via Spark UI and logs.
• Familiarity with Parquet/ORC and data serialization formats and their trade-offs.

Responsibilities

• Design, develop, and maintain distributed data pipelines using Spark (Core, SQL, Streaming) in Scala.
• Build ingestion and transformation workflows across Hadoop ecosystem for TB-PB scale data.
• Tune Spark jobs with partitioning, caching, joins, and memory sizing to optimize performance.
• Implement real-time/near-real-time ingestion with NiFi and/or Kafka.
• Embed data quality controls within pipelines and data lineage documentation.
• Write and optimize HiveQL and Spark SQL for curated and consumption layers.
• Automate orchestration with Airflow, Oozie, or Control-M.
• Participate in code reviews, CI/CD, testing, and production support.
• Troubleshoot failures, SLA breaches, and performance regressions; drive root-cause fixes.
• Document data flows, lineage, and operational runbooks.

Skills

Spark (Scala)

Hadoop

SQL

Data pipelines

Python/PySpark

CI/CD tooling

Linux scripting

Education

Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline

Tools

HDFS

Hive

YARN

MapReduce

Parquet/ORC

Avro

Git

Maven

SBT

Jenkins

Job description

Infinite Computer Solutions is seeking an experienced Big Data Engineer to design, build, and optimize large-scale batch and streaming data pipelines on the Hadoop ecosystem using Apache Spark and Scala.

You will work with data architects and analytics teams to ingest, transform, and enrich clickstream, network, and location datasets, ensuring data quality and reliability in production environments.