Data Engineer

The Judge Group; New York (NY); On-site; USD 150,000 - 190,000; Full time

Benefits offered by this job

Job summary

The Judge Group is seeking a Lead Software Engineer – Data Engineering in New York, NY to design and build scalable, cloud-native data platforms and pipelines. You will lead the development of robust data solutions, drive architectural decisions, and mentor engineers while ensuring reliability, security, and operational excellence.

You will design large-scale data pipelines, implement AWS-based data solutions with Spark, lead data modelling and warehousing initiatives, and collaborate with

Qualifications

• Proficiency in Python and/or Java with Spark/PySpark.

Responsibilities

• Design and build large-scale data pipelines and platforms.
• Develop cloud-native solutions using AWS services and Spark.
• Lead data modelling, warehousing, migration, and analytics initiatives.
• Improve system performance, resilience, monitoring, and automation.
• Collaborate with cross-functional teams and mentor junior engineers.

Skills

Python

Java

Spark

PySpark

Data Pipelines

Data Modelling

Data Warehousing

Data Migration

AWS

Kubernetes

CI/CD

Agile methodologies

REST APIs

Linux scripting

JIRA

Tools

Airflow

Glue

EMR

S3

Athena

Kinesis

MSK

Lake Formation

Job description

About the Role

We are seeking a Lead Software Engineer – Data Engineering to design and build scalable, cloud‑native data platforms and pipelines. You will lead the development of robust data solutions, drive architectural decisions, and mentor engineers while ensuring reliability, security, and operational excellence.

Responsibilities

• Design and develop large‑scale data pipelines and data platforms.
• Build cloud‑native solutions using AWS services and Spark.
• Lead data modelling, warehousing, migration, and analytics initiatives.
• Improve system performance, resilience, monitoring, and automation.
• Collaborate with cross‑functional teams and mentor junior engineers.

Required Qualifications

• Strong proficiency in Python and/or Java with Spark/PySpark.
• Experience with Data Pipelines, Data Modelling, Data Warehousing, and Data Migration.
• Hands‑on experience with AWS (Glue, EMR, S3, Athena, Kinesis/MSK, Airflow, Lambda, Step Functions, Lake Formation).
• Knowledge of Kubernetes, CI/CD, Agile methodologies, and software design principles.
• Experience with data formats such as Parquet, Iceberg, Avro, or JSON.
• Understanding of AWS security concepts including IAM, KMS, Encryption, and Secrets Manager.

Preferred Qualifications

• Experience building REST APIs using Java.
• Knowledge of Hybrid Cloud architectures.
• Exposure to Jenkins, Git, Linux Scripting, JIRA, and TDD.
• Strong leadership, problem‑solving, and communication skills.