Staff Engineer - Payments & Commerce Platform

OpenKrill; United States; Vor Ort; USD 180.000 - 240.000; Vollzeit

Zusammenfassung

HighLevel is seeking a Principal Engineer to steer domain modeling, architecture, and reliability for our global commerce platform. You will shape core services—subscriptions, payments, catalog, pricing, inventory, and tax—driving correctness, auditability, and performance at scale.

This IC role (no direct reports) focuses on designing systems, establishing standards, and growing engineers across teams, with strong governance and cross-functional collaboration.

Qualifikationen

• 10+ years building and operating backend systems.
• 5+ years in Go on critical paths.
• Proficiency with protobuf + ConnectRPC/gRPC and API lifecycle management.
• Distributed systems fundamentals: idempotency, ordering, and scalability.
• Event-driven architectures on GCP (Pub/Sub) and data stores.
• Kubernetes/GKE operations at scale and CI/CD.
• Security, PCI/compliance basics, and data governance.
• Strong communication and design documentation skills.

Aufgaben

• Architect and ship multi-tenant, planet-scale services with clear domain boundaries and hard SLOs.
• Own API design, versioning, and contract testing to ensure long-term compatibility.
• Guarantee resilience of core payment paths with timeouts, retries, and circuit breakers.
• Ensure auditability with immutable event streams and traceable data lineage.
• Define and enforce service catalog, runbooks, and deployment standards.
• Mentor engineers and lead design reviews, ADRs/RFCs, and engineering best practices.

Kenntnisse

Go programming

Backend systems

Distributed systems

API lifecycle

Leadership

Tools

protobuf + ConnectRPC

gRPC

MongoDB

Firestore

ClickHouse

GKE / Kubernetes

Pub/Sub

Redis

OpenTelemetry

Jobbeschreibung

About HighLevel:HighLevel is an AI-powered business operating system that gives agencies, entrepreneurs and SMBs the infrastructure to build, automate and scale. Today, HighLevel supports SMBs across 150+ countries, fueling community-driven growth rooted in real customer outcomes.To date, businesses operating on HighLevel have generated over $7 billion in ecosystem value, demonstrating the impact of shared infrastructure at scale. By centralizing conversations, automation and intelligence into one system, we help businesses move faster, reduce complexity and execute efficiently.Behind the platform, HighLevel powers more than 4 billion API hits and 2.5 billion message events daily. With 250 terabytes of distributed data, 250+ microservices and over 1 million domain names supported, our architecture is built for performance, resilience and long-term scalability.

Our PeopleWith over 2,000 team members across 10+ countries, HighLevel operates as a global, remote-first organization built for speed and ownership. We value initiative, clarity and execution, creating space for ambitious people to build systems that support millions of businesses worldwide. Here, innovation thrives, ideas are celebrated and people come first, no matter where they call home.

Our ImpactEvery month, HighLevel enables more than 1.5 billion messages, 200 million leads and 20 million conversations for the more than 1 million businesses we support. Behind those numbers are real people building independence, expanding opportunity and creating measurable impact. We're proud to be a part of that.Learn more about us on our YouTube Channel or Blog Posts.

About the Role:

We’re building a global commerce platform to power $1T+ in annual transactions for millions of SMEs. We want our Principal Enggs to be the technical force‑multiplier who sets the domain modelling, architecture, raises the reliability bar, and multiplies team effectiveness. You’ll steward our core commerce models—subscriptions, payments , Catalog, Pricing, Inventory, fulfillment, reconciliation , tax —and the services around them, ensuring correctness by design, auditability, and delightful performance at scale. This is an IC role with org‑level influence (no direct reports), focused on designing systems, shaping standards, and growing engineers.

Tech Stack:

Backend: Go, ConnectRPCDatabases: MongoDB, Firestore, ClickHouseCloud: GCP (GKE), Pub/Sub, Redis, OpenTelemetry

What You’ll Do:

• Architect and ship multi‑tenant, planet‑scale services (checkout, subscriptions, payments orchestration, invoicing, tax hooks) with clear domain boundaries (DDD) and hard SLOs
• Be the custodian of API & schema design: own protobuf/ConnectRPC conventions, versioning policy, deprecation playbooks, and Buf breaking‑change checks—so our contracts stand the test of time
• Guarantee resilience & availability of core payment paths: timeouts, retries with jitter, circuit breakers, idempotency keys, outbox/Saga patterns, hedged requests, and graceful degradation
• Ensure complete auditability: append‑only double‑entry ledger, immutable event streams, trace‑linked entities (OTel trace/span IDs), tamper‑evident trails, and reconciliations that tie out to the cent
• Own error boundaries end‑to‑end: enumerate failure domains (PSP, network, data, concurrency, quota, browser, device); design uniform error contracts; implement compensations/backfills and automated replay
• Keep track of every deployed thing: services, workers, triggers, cron, subscriptions—own the service catalog and scorecards (owners, SLOs, runbooks, PDBs, HPA/VPA, budgets, quotas, timeouts)
• Configuration & limits stewardship: enforce sane defaults across GKE, Pub/Sub, Redis, Firestore/Mongo, ClickHouse—connection pools, ack deadlines, batch sizes, TTLs, memory/FD limits, and GCP quotas
• Observability as a product: pervasive OpenTelemetry, RED/USE metrics, exemplars, trace sampling, SLO dashboards, and alerting that wakes humans only for user‑impacting issues
• Production excellence: canary/blue‑green rollouts, automated rollbacks, chaos drills, DR playbooks (RPO/RTO), multi‑region failover strategies, and incident command on rotation
• Security & compliance by design: PCI scope minimization, tokenization/vaulting, secrets/KMS hygiene, data retention/archival, and privacy controls—embed checks in CI/CD
• Developer acceleration: pave golden paths (service templates, ADR/RFC process, linting/formatting, contract tests, eternal envs, load/perf harnesses) to make the right thing the easy thing

What You’ll Lead:

• Core domain evolution: orchestration → ledger → reconciliation flows with crisp invariants and consistency guarantees (read‑your‑writes where needed, eventual where appropriate)
• Reliability strategy: SLIs/SLOs, error budgets, capacity planning, cost/FinOps guardrails, multi‑region posture, and DR exercises
• API & data governance: canonical models, schema lifecycle (compatibility matrix, migrations), data lifecycle (retention, archival, compliance)
• Practice leadership for HighLevel: design reviews, postmortems, technical strategy, coding standards, and mentorship across teams—raise the bar for the org
• Hiring & team growth: help us hire, scale, and train the right team; shape interview loops, rubrics, onboarding, and ongoing learning (brown bags, reviews, pair design)
• Cross‑functional partnership: collaborate with Product/Marketing/Support to translate platform capabilities and constraints into roadmaps, GTM narratives, and reliable customer outcomes
• Risk & roadmap: maintain a technical risk register, make build‑vs‑buy calls, and propose simplifications or deprecations that meaningfully reduce complexity and MTTR

Minimum Qualifications:

• 10+ years building and operating backend systems (at least 5+ years in Go), with 2–3+ years acting as a Staff/Principal‑level IC or Tech Lead for critical paths
• Deep proficiency with protobuf + ConnectRPC/gRPC and API lifecycle management (versioning, compatibility, contract testing, Buf)
• Distributed systems fundamentals: idempotency, exactly‑once‑ish via dedupe/outbox, ordering, consensus basics, backpressure, concurrency control
• Event‑driven architectures on GCP (Pub/Sub), plus Redis for fast paths; strong schema design in MongoDB/Firestore and analytics/reporting patterns on ClickHouse
• Kubernetes/GKE operations at scale: autoscaling (HPA/VPA), PDBs, resource limits/requests, multi‑region topologies, CI/CD, canary/blue‑green
• Reliability engineering: SLIs/SLOs, error budgets, capacity & load testing, incident management, DR/BCP
• Security & compliance: secrets/KMS best practices, PCI basics (scope reduction, key rotation), and data governance (retention/archival)
• Testing discipline: unit, integration, contract, property‑based, performance; test data management and deterministic environments
• Frontend collaboration: solid understanding of Vue.js + TanStack Query to shape clean API surfaces and performance budgets across the boundary
• Exceptional technical writing & communication: design docs, ADRs/RFCs, postmortems, and stakeholder updates

Nice to Have:

• Hands‑on integrations with major PSPs/local rails (e.g., UPI, wallets, BNPL, cards/3DS2) and reconciliation at scale
• Experience with active‑active or multi‑region designs; chaos engineering; traffic management
• Observability leadership with OpenTelemetry at org scale (tail‑based sampling, exemplars)
• FinOps experience: cost baselining, quotas, budget alarms, and workload right‑sizing
• Familiarity with regulatory frameworks (PCI DSS, SOC 2/ISO 27001) and privacy laws relevant to our markets

EEO Statement:

The company is an Equal Opportunity Employer. As an employer subject to affirmative action regulations, we invite you to voluntarily provide the following demographic information. This information is used solely for compliance with government recordkeeping, reporting, and other legal requirements. Providing this info