Senior Principal Performance Engineer

The Depository Trust & Clearing Corporation (DTCC); Jersey City (NJ); Hybrid; USD 170,000 - 210,000; Full time

Benefits offered by this job

Competitive compensation

Health and life insurance

Pension / Retirement benefits

Paid Time Off

Hybrid work model: 3 days onsite, 2 in

Leaves of absence for wellbeing

Job summary

DTCC is seeking a Senior Principal Performance & Resiliency Engineer to lead the strategy, architecture, and execution of performance engineering, resilience testing, capacity planning, and operational readiness across complex distributed systems and cloud-native environments.

You will partner with engineering, infrastructure, architecture, and product teams to proactively identify bottlenecks, reduce systemic risks, and enhance platform reliability.

Qualifications

• 10+ years of hands-on performance engineering and reliability experience in latency-sensitive or high-volume transaction systems.
• Strong discipline across load, stress, soak and spike testing with results analyzed by p99/p99.9.
• Expertise across JVM tuning, heap/thread dumps, flame graphs, and bottleneck isolation.

Responsibilities

• Make NFRs measurable and release-gated with clear latency, throughput, availability and capacity targets.
• Embed continuous performance and resilience validation in CI to catch issues before production.
• Establish estate-wide observability standards, SLIs, and capacity planning baselines.

Skills

Performance engineering

Reliability engineering

Chaos engineering

Observability

OpenTelemetry

JVM tuning

Flame graphs

Distributed systems

Education

Bachelor's degree

Tools

JMeter

Gatling

k6

Prometheus

Grafana

Dynatrace

Datadog

Splunk

Job description

Are you ready to make an impact at DTCC? Do you want to work on innovative projects, collaborate with a dynamic and supportive team, and receive investment in your professional development? At DTCC, we are at the forefront of innovation in the financial markets. We are committed to helping our employees grow and succeed. We believe that you have the skills and drive to make a real impact. We foster a thriving internal community and are committed to creating a workplace that looks like the world that we serve.

The Information Technology group delivers secure, reliable technology solutions that enable DTCC to be the trusted infrastructure of the global capital markets. The team delivers high-quality information through activities that include development of essential, building infrastructure capabilities to meet client needs and implementing data standards and governance.

Pay And Benefits

• Competitive compensation, including base pay and annual incentive
• Comprehensive health and life insurance and well-being benefits, based on location
• Pension / Retirement benefits
• Paid Time Off and Personal/Family Care, and other leaves of absence when needed to support your physical, financial, and emotional well-being.
• DTCC offers a flexible/hybrid model of 3 days onsite and 2 days remote (onsite Tuesdays, Wednesdays and a third day unique to each team or employee).

The Impact you will have in this role

As a Senior Principal Performance & Resiliency Engineer, you will play a critical role in ensuring the reliability, scalability, and operational excellence of DTCC's most business-critical platforms. You will lead the strategy, architecture, and execution of performance engineering, resilience testing, capacity planning, and operational readiness initiatives across complex distributed systems and cloud-native environments. In this role, you will partner closely with engineering, infrastructure, architecture, and product teams to proactively identify performance bottlenecks, eliminate systemic risks, and enhance platform resilience. Your leadership will help ensure that critical financial market infrastructure systems can meet evolving business demands, regulatory expectations, and client service commitments while maintaining the highest standards of availability and reliability. You will drive a culture of engineering excellence by championing observability, automation, chaos engineering, disaster recovery preparedness, and continuous performance optimization. Your contributions will directly strengthen DTCC's ability to deliver secure, resilient, and highly available services that support global financial markets.

Your Primary Responsibilities

• Make NFRs measurable and release-gated. Define clear targets for latency, throughput, availability, recovery, and capacity, and validate them under production-like volume before release.
• Embed continuous performance and resilience validation. Automate regression testing and failure-injection scenarios in CI so reliability issues are identified before production impact.
• Establish estate-wide observability and readiness standards. Create a common instrumentation, SLI, and capacity-planning baseline so risk platforms can scale, recover, and be operated consistently across the estate.

Qualifications

• Minimum of 10 years of related experience
• Bachelor's degree preferred or equivalent experience

Talents Needed For Success

• 10+ years of hands-on performance engineering and reliability experience in latency-sensitive or high-volume transaction systems, with continued depth in profiling, dump analysis, and test harness development.
• Strong performance engineering discipline across load, stress, soak, and spike testing, using workload models grounded in real production traffic and analyzing results through p99 and p99.9 behavior rather than averages.
• Deep diagnostic capability across JVM tuning, heap and thread dump analysis, flame graphs, async profiling, and bottleneck isolation across application, database, network, and storage layers—even when standard dashboards appear healthy.
• Resiliency and chaos engineering expertise including fault injection, failure-mode cataloging, game day design, and DR exercises supported by measured RTO/RPO evidence.
• Distributed systems and data platform performance depth spanning Kafka throughput, consumer lag, rebalancing, partition skew, backpressure, exactly-once tradeoffs, object storage, lakehouse query performance, and Snowflake warehouse tuning.
• Observability architecture experience covering OpenTelemetry, distributed tracing, metrics taxonomy, SLI/SLO and error budget design, actionable alerting, and disciplined control of cardinality and telemetry cost.
• Automation embedded into delivery pipelines through CI performance gates, automated baselining, regression detection, and practical test data and environment strategies for constrained end-to-end environments.
• Broad tooling proficiency with engineering depth across JMeter, Gatling or k6; Prometheus and Grafana; Dynatrace, Datadog or Splunk; and the ability to build custom harnesses when standard tools do not fit the workload.
• Regulated environment fluency including DR obligations, production readiness reviews, audit evidence, and disciplined incident and postmortem practices.
• Influence without authority with the credibility to set and enforce NFR standards across delivery squads and support release-readiness decisions with clear, evidence-based judgment.

The salary range is indicative for roles at the same level within DTCC across all US locations. Actual salary is determined based based on the role, location, individual experience, skills, and other considerations. We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, sex, gender, gender expression, sexual orientation, age, marital status, veteran status, or disability status. We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, to perform essential job functions, and to receive other benefits and privileges of employment. Please contact us to request accommodation.