Lead Associate Principal, Software Engineering: Software Development Test (SDET)

theocc; United States; On-site; USD 150,000 - 230,000; Full time

Job summary

theocc is seeking an Associate Principal for Performance Testing within OCC's Enterprise Quality Engineering organization. You will design and execute comprehensive performance and failover testing strategies across critical financial systems, ensuring RTO/RPO targets are met in cloud and multi-region deployments.

You will lead test planning, execute tests with LoadRunner/JMeter/Gatling, and build automation, pipelines, and observability integrations to support regulatory audits and continuous

Qualifications

• 7+ years hands-on experience in performance testing and engineering.
• 5+ years designing and executing failover/DR testing across multi-region architectures (cloud and hybrid environments).
• Strong expertise in performance testing tools: LoadRunner, JMeter, Gatling.
• Proficiency in scripting/programming: Python, Java, JavaScript for test frameworks.
• Solid experience with AWS cloud environments (EC2, RDS, Kafka) and performance monitoring.
• Working knowledge of Kubernetes and containerized performance testing (Docker, Kafka).
• Experience with CI/CD pipelines (Jenkins, GitHub, GitLab CI) and integrating tests into automation.
• Proficiency with APM/observability platforms and correlating test results with production telemetry.

Responsibilities

• Define test plans, load profiles, and ramp scenarios for high-volume systems.
• Execute tests using LoadRunner, JMeter, and Gatling; analyze results and optimize.
• Lead DR testing across primary, secondary, and tertiary regions.
• Build and maintain performance test automation using Python and Java.
• Develop CI/CD pipelines for continuous performance validation.
• Implement regression testing to catch performance regressions.
• Validate behavior across multi-region deployments and data integrity during failover.
• Define and validate RTO/RPO targets for critical apps.
• Design failover scenarios and coordinate with infra teams.

Skills

Performance testing

Failover/DR testing

CI/CD integration

Python/Java/JavaScript

LoadRunner

JMeter

Gatling

AWS

Kubernetes

SQL

Tools

LoadRunner

JMeter

Gatling

Python

Java

Jenkins

GitHub Actions

GitLab CI

Postman

Job description

What You'll Do:

OCC's Enterprise Quality Engineering organization seeks an Associate Principal in Performance Testing to design and execute comprehensive performance and failover testing strategies across mission-critical clearing, risk, and settlement systems. This role combines hands-on performance engineering expertise to ensure systems meet regulatory RTO/RPO requirements, handling multi-region failover scenarios, cloud-native architectures, and distributed event-driven systems.

Primary Duties and Responsibilities:

• Define test plans, load profiles, and ramp scenarios for high-volume transaction systems
• Execute tests using LoadRunner, JMeter, and Gatling; analyze results and recommend optimizations
• Lead DR testing across primary, secondary, and tertiary regions
• Build and maintain performance test automation using Python, Java, and open-source tools
• Develop CI/CD-integrated pipelines (Jenkins, GitHub Actions) for continuous performance validation
• Implement regression testing to catch performance degradation from code changes
• Validate application behavior across multi-region deployments
• Define and validate RTO/RPO targets for critical applications
• Design failover scenarios (region switching, data replication, automation, post-failover validation)
• Partner with infrastructure/application teams to verify state consistency and data integrity during failover
• Test replication lag, failover triggers, and cross-region data consistency
• Validate Kafka producer/consumer behavior under failover conditions
• Confirm performance SLAs hold during region-switching scenarios
• Integrate test results with observability platforms
• Maintain and evolve LoadRunner scripts and environments
• Document methodologies, baselines, and trend analysis
• Mentor QE engineers on performance and failover testing techniques
• Collaborate with development, DevOps, and infrastructure teams on requirements and capacity planning
• Ensure testing aligns with PGP compliance and financial market infrastructure regulations
• Document test evidence to support regulatory audits
• Validate performance SLAs meet regulatory RTO/RPO requirements

Supervisory Responsibilities:

None

Qualifications:

The requirements listed are representative of the knowledge, skill, and/or ability required. Reasonable accommodations may be made to enable individuals with disabilities to perform the primary functions.

• 7+ years hands-on experience in performance testing and engineering, including design and execution of complex performance test scenarios
• 5+ years designing and executing failover/DR testing across multi-region architectures (cloud and hybrid environments)
• Strong expertise in performance testing tools: LoadRunner (enterprise scaling), JMeter, Gatling, with ability to evaluate tool trade-offs
• Proficiency in scripting/programming: Python, Java, JavaScript, with ability to develop performance testing frameworks
• Solid hands-on experience with AWS cloud environments, including EC2, RDS, Kafka, and performance monitoring
• Working knowledge of Kubernetes and containerized application performance testing (Docker, Kafka, event-driven systems)
• Experience with UC4 (Automic) job scheduling in test execution and orchestration
• Hands-on experience testing Kafka producer/consumer performance, lag validation, and message throughput
• Strong understanding of relational databases (SQL Server, Oracle, DB2) and ability to write queries for performance analysis
• Experience with CI/CD pipelines (Jenkins, GitHub, GitLab CI) and integration of performance tests into automation
• Proficiency with APM/observability platforms (Splunk, Dynatrace, AppDynamics) and correlation of test results with production telemetry
• Working knowledge of Linux operating systems and command-line tools for performance analysis
• Experience with test management tools (Jira, Zephyr, SpiraTest) and defect tracking integration
• Background in finance domain OR highly regulated, low-latency, high-availability systems (clearing, settlement, risk)
• Experience with multi-region testing (primary/secondary/tertiary architectures) with performance validation across regions

Technical Skills:

• Familiarity with AI/ML-augmented testing tools for automated test case generation, self-healing automation, and predictive defect analysis.
• Experience with API and contract testing (Postman, RestAssured, Pact) and service virtualization for testing complex distributed and microservices architectures.

Understanding of

Understanding of