Lead Software Engineer- Financial Services Data Engineering: Pyspark / Java / BigData / Datalake / AI

ARCHER; Jersey City (NJ); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Health Care Coverage

On-Site Health And Wellness Centers

Retirement Savings Plan

Backup Childcare

Tuition Reimbursement

Mental Health Support

Financial Coaching

Job summary

ARCHER is seeking a senior expert to lead modernization of analytics and data engineering on our strategic data mesh. You will define an AI-driven data product lifecycle and build human-in-the-loop agents for data quality, schema changes, incident triage, and governance.

You will provide technical leadership, improve engineering practices with approved AI-assisted tools, and deliver scalable analytics platforms with strong monitoring, lineage, and regulatory controls.

Qualifications

• Requires 5+ years of applied software engineering leadership delivering AI-first analytics and data engineering at scale.
• Hands-on experience with modern big data platforms and Python/PySpark or Java.
• Expertise in AI/ML, data governance, CI/CD, secure AI-assisted development, and the software development lifecycle.

Responsibilities

• Lead modernization of analytics and data engineering on the data mesh.
• Define an AI-driven data product lifecycle and build human-in-the-loop agents for data quality, schema changes, incident triage, and governance.
• Provide technical leadership and improve engineering practices with AI-assisted tools; deliver scalable analytics platforms with strong monitoring and lineage.

Skills

Python

PySpark

Java

Machine Learning

Natural Language Processing

Large Language Models

Data Governance

Data Mesh

Data Lakehouse

AI-first analytics

CI/CD

Tools

Apache Spark

Databricks

Snowflake

Apache Iceberg

CI/CD (tooling)

Job description

Lead modernization of analytics and data engineering on the strategic data mesh, defining an AI-driven data product lifecycle and building human-in-the-loop agents for data quality, schema changes, incident triage, and governance. Provide technical leadership, improve engineering practices with approved AI-assisted tools, and deliver scalable analytics platforms with strong monitoring, lineage, and regulatory controls.

Requirements: Requires at least five years of applied software engineering experience, with leadership delivering AI-first analytics and data engineering at scale. Candidates should have hands‑on experience with modern big data platforms and Python/PySpark or Java, plus expertise in AI/ML, data governance, CI/CD, secure AI-assisted development, and the software development lifecycle.

Key Skills: Python, PySpark, Java, Apache Spark, Databricks, Snowflake, Apache Iceberg, Big Data Engineering, Data Mesh, Data Lakehouse, Machine Learning, Natural Language Processing, Large Language Models, Agentic Frameworks, Data Governance, CI/CD

Benefits: Health Care Coverage, On-Site Health And Wellness Centers, Retirement Savings Plan, Backup Childcare, Tuition Reimbursement, Mental Health Support, Financial Coaching