Apache Spark

TechDigital Group; Jersey City (NJ); On-site; USD 120,000 - 190,000; Full time

Job summary

TechDigital Group in Jersey City, NJ is seeking a seasoned data engineer to design and optimize streaming data pipelines. You will leverage Apache Spark and Kafka to build scalable ETL processes and analytics-ready datasets.

The role requires deep knowledge of data modeling, Parquet, and cloud platforms, with hands-on experience in Hadoop ecosystems. Strong communication and problem-solving skills are essential.

Qualifications

• 5+ years of experience with Apache Spark (including Structured Streaming).
• Strong proficiency in Scala, with ability to write clean, performant code.
• 3+ years of experience with Apache Kafka for real-time streaming and messaging.
• Solid understanding of data modeling, ETL, and data pipeline orchestration.
• Familiarity with Hadoop ecosystem, Hive, Parquet, or similar.
• Experience with CI/CD, Docker, and Kubernetes is a plus.
• Exposure to cloud platforms like AWS (EMR, MSK), Azure, or GCP is preferred.
• Excellent problem-solving, communication, and analytical skills.

Skills

Apache Spark

Scala

Apache Kafka

Data Modeling

ETL

Data Pipeline Orchestration

Tools

Hadoop

Hive

Parquet

CI/CD

Docker

Kubernetes

AWS

Azure

GCP

Job description

• 5+ years of experience working with Apache Spark (including Structured Streaming).
• Strong proficiency in Scala, with ability to write clean, performant code.
• 3+ years of experience with Apache Kafka for real-time streaming and messaging.
• Solid understanding of data modeling, ETL, and data pipeline orchestration.
• Familiarity with Hadoop ecosystem, Hive, Parquet, or similar.
• Experience with CI/CD, Docker, and Kubernetes is a plus.
• Exposure to cloud platforms like AWS (EMR, MSK), Azure, or GCP is preferred.
• Excellent problem-solving, communication, and analytical skills.