Senior Product Engineer

ACA Group; Durham (NC); On-site; USD 110,000 - 140,000; Full time

Job summary

ACA Group is looking for a Senior Algorithm Developer located in Durham, NC, to validate and optimize their PySpark-based surveillance platform. This role blends deep expertise in PySpark/Python and SQL with knowledge of financial markets and regulatory frameworks. The ideal candidate will have over 5 years of experience in software development, particularly in data processing systems. The position offers significant responsibilities in ensuring production quality and compliance with regulatory standards.

Qualifications

• 3+ years of hands-on PySpark/Python development in production environments.
• 5+ years of advanced SQL experience, including performance tuning.
• Strong AWS experience with S3, EMR, IAM, CloudWatch.

Responsibilities

• Validate and QA migrated surveillance algorithms by reconciling outputs.
• Optimize PySpark algorithms to meet processing SLAs across large data volumes.
• Enhance and develop new algorithms aligned to evolving regulatory requirements.

Skills

PySpark/Python development

Advanced SQL

Spark expertise

AWS

CI/CD and Git workflows

Financial instruments knowledge

Job description

Senior Algorithm Developer – PySpark / SQL

ACA Global delivers market abuse surveillance solutions to institutional investors, hedge funds, and asset managers. The Market Abuse Surveillance (MAS) platform monitors trading activity across equities, fixed income, and derivatives to identify potential regulatory and policy violations.

The Opportunity

This role sits on the Market Abuse Surveillance Product Development team and focuses on validating, optimizing, and advancing a PySpark‑based surveillance platform. Following a multi‑year algorithm migration initiative, the emphasis is now on production quality, performance optimization, and next‑generation algorithm development. The position blends deep PySpark/Python and SQL expertise with financial markets and regulatory domain knowledge.

What You'll Do

• Validate and QA migrated surveillance algorithms by reconciling PySpark outputs against legacy SQL baselines across multiple client environments
• Optimize PySpark algorithms and Spark infrastructure to meet processing SLAs across large data volumes
• Enhance and develop new surveillance algorithms aligned to evolving regulatory and client requirements
• Support legacy SQL systems during transition and ensure data integrity and auditability

Technical Skill Set

• 3+ years of hands‑on PySpark/Python development in production environments
• Strong Spark expertise (DataFrames, SQL functions, windowing, joins, optimization, execution plans)
• 5+ years of advanced SQL experience (CTEs, window functions, performance tuning, stored procedures)
• Experience operating within the Spark ecosystem (EMR, Iceberg, Spark UI)
• Strong AWS experience (S3, EMR, IAM, CloudWatch, cost optimization)
• Familiar with CI/CD, Git workflows, JIRA, automated testing, and code review practices

Domain Expertise (or Strong Willingness to Learn)

• Financial instruments: equities, fixed income, and derivatives
• Trade lifecycle and portfolio concepts
• Market abuse and regulatory surveillance (KYC/AML, best execution, fair dealing)
• Institutional trading behavior and surveillance patterns

Experience Profile

• 5+ years total software development experience, including large‑scale data processing systems
• Proven track record delivering production‑ready data or algorithmic platforms
• Strong validation, testing, and reconciliation experience across complex data systems
• Background in fintech, RegTech, or financial services strongly preferred