Senior PySpark and Big Data Lead

Citi; Jersey City (NJ); On-site; USD 128,000 - 213,000; Full time

Job summary

Citi in Jersey City, NJ is seeking a senior Big Data PySpark Lead Engineer to architect and implement advanced data ingestion and processing systems. You will guide teams through complex data workflows, ensure architectural alignment, and drive reliable, scalable solutions.

Responsibilities include building PySpark pipelines, leveraging Hive/HDFS/Sqoop/Spark/Impala/Scala, and ensuring low-latency streaming and batch processing.

Qualifications

• Hands-on PySpark and Big Data processing experience in production
• Strong Hadoop ecosystem knowledge (Hive, HDFS, Sqoop, Spark, Impala, Scala)
• Proficient in complex SQL development for data analysis across large datasets
• Understanding of distributed systems architecture and data warehouse concepts
• Experience with data modeling and dimensional modeling techniques
• Scripting and job scheduling with Autosys or similar tools
• Excellent communication and ability to convey technical concepts to diverse audiences

Responsibilities

• Build and maintain scalable data pipelines using PySpark in a Big Data environment
• Design and develop solutions across Hadoop ecosystem for data ingestion, processing and storage
• Develop and manage real-time and batch data workflows with low latency
• Write complex SQL queries to extract, validate, and analyze data
• Design data models and architectures aligned with data warehouse principles
• Automate pipeline scheduling and orchestration using shell scripting and Autosys
• Identify and resolve technical risks and data issues to maintain data platform integrity

Skills

PySpark

Big Data processing

SQL queries

Distributed systems

Data modeling

Leadership

Communication skills

Education

Bachelor’s degree

Master’s degree preferred

Tools

Hive

HDFS

Sqoop

Spark

Impala

Scala

Autosys

Job description

Citi in Jersey City, NJ is seeking a senior Big Data PySpark Lead Engineer to architect and implement advanced data ingestion and processing systems. You will guide teams through complex data workflows, ensure architectural alignment, and drive reliable, scalable solutions.

Responsibilities include building PySpark pipelines, leveraging Hive/HDFS/Sqoop/Spark/Impala/Scala, and ensuring low-latency streaming and batch processing.