Senior Developer Hadoop

Centraprise; Pennington (NJ); On-site; USD 140,000 - 190,000; Full time

Job summary

Centraprise in Jersey City or Pennington, NJ is seeking a seasoned Data Engineer/Architect with 10+ years of experience to lead surveillance data processing platforms on Hadoop and Spark. This onsite role requires deep PySpark expertise, knowledge of HDFS/Hive and strong communication to influence across teams.

You will define target state architecture, drive end-to-end design, ensure regulatory traceability, and mentor senior engineers while aligning with enterprise standards.

Qualifications

• Deep Expertise in PySpark with performance tuning
• Strong knowledge of Hadoop ecosystem (HDFS, Hive/Impala, YARN)
• Experience designing and governing enterprise data platforms
• Experience building data lakes, ELT/ETL pipelines, batch and real-time processing
• Proficiency in Java, Scala, and SQL
• Experience in financial services surveillance/regulatory tech
• Experience supporting regulatory or audit facing platforms
• Knowledge of data governance, lineage, and data quality controls

Responsibilities

• Define target state architecture and roadmap for surveillance data processing platforms using Hadoop and Spark
• Establish standard patterns for alerting, enrichment, aggregation, and reporting pipelines
• Review and approve technical designs aligning with regulatory requirements
• Provide hands-on architectural guidance during development, testing, and deployment
• Modernize large-scale Hadoop surveillance workloads to improve performance and control coverage
• Ensure audit readiness, lineage, and traceability in data flows
• Mentor engineers and lead architecture review forums

Skills

PySpark

Data engineering

Hadoop ecosystem

Spark Structured streaming

Python

Java/Scala

Data governance & lineage

Regulatory/compliance domain

Communication skills

Architectural leadership

Education

Bachelor’s or Master’s degree in Computer Science or related field

Tools

Airflow

Oozie

Control-M

Hadoop

Spark

Kubernetes

Job description

Job Location : Jersey City, NJ or Pennington, NJ (ONSITE)

Job Type : Full-Time

Job Description:

Must Have Technical/Functional Skills

Primary Skill: Data Engineering, Platform Engineering or architecture roles. Deep Expertise in Pyspark.

Experience: 10+ years

Bachelor’s or master’s degree in computer science or related field.

• Required Hard and Soft Skills / Experience
• Deep Expertise in PySpark, including performance tuning and optimization
• Solid knowledge of Hadoop ecosystem (HDFS,Hive/Impala, YARN)
• Proven experience designing and governing enterprise, regulatory facing data platforms.
• Expertise in designing data lakes, ELT/ETL pipelines, batch and real time data processing solution
• Proficiency in programming languages such as Java, Scala and SQL
• Strong understanding of non-functional requirements and production support models
• Clear written and verbal communication skills with ability to influence across organizations
• Financial services experience, particularly in Market Surveillance, AML, Fraud, Or Risk Technology
• Experience supporting regulatory or audit facing platforms
• Kafka and Spark Structured streaming exposure
• Familiarity with Orchestration tools(Airflow,Control-M,Oozie)
• Knowledge of data governance, lineage, and data quality controls

Daily Responsibilities/Priorities

Enterprise & Surveillance Architecture

Define target state architecture and strategic roadmap for surveillance data processing platforms leveraging Hadoop,Spark, Pyspark and Python

Establish standard architectural patterns for alert generation, enrichment, aggregation and reporting pipelines.

Ensure architecture aligns with enterprise technology standards and Surveillance control expectation

Solution Design and Delivery Support

Translate surveillance business requirements(e.g, market misconduct detection, regulatory coverage) into scalable technical designs

Review and approve detailed technical designs, ensuring alignment with functional intent, regulatory requirements, and architectural standards.

Provide hands‑on architectural guidance to engineering teams during development, testing and implementation

Design and standardization of Spark/Pyspark frameworks supporting surveillance alert generation and enrichment

Modernization and Optimization of large‑scale Hadoop surveillance workloads to improve performance, stability and control coverage

Implementation of enterprise-consistent architecture patterns enabling audit readiness, lineage, and regulatory traceability

Support end‑to‑end delivery across the SDLC, minimizing rework and technical debt

Architecture Governance & Leadership

Participate in and lead architecture review forums and design walkthroughs

Mentor senior engineers and promote adoption of Standard framework and best practices

Influence enterprise surveillance platform strategy through architecture governance

Maintain high‑quality documentation, including business requirement, data definitions and process flows

Proactively communicate risks, dependencies and potential timeline impact to stakeholders

Non-Functional requirement and product Readiness.