Principal Engineer: Stream Governance & Metadata

IBM; San Diego (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

IBM Software in San Diego is seeking a Principal Engineer to lead the architecture and delivery of large-scale governance systems for streaming data.

You will own core domains, mentor engineers, and collaborate with product leadership to balance technical depth with business impact, ensuring reliability and scale across Confluent‑like platforms.

This role requires autonomy in ambiguous spaces and a track record of shaping technology strategy across multiple teams.

Qualifications

• Experience leading the architecture and delivery of large-scale distributed systems.
• Knowledge of fault tolerance, consistency, scalability, and cloud-native platforms.
• Experience with schema management, data catalogs, or metadata/governance systems.
• Ability to navigate ambiguity and influence stakeholders at the executive level.
• Strong track record of delivering trusted streaming governance solutions.

Responsibilities

• Define and drive company-level technical strategy for governance systems, ensuring long-term innovation for schema and metadata infrastructure.
• Architect and evolve high-performance metadata management systems with low-latency lookups and strong consistency across streaming and processing engines.
• Design and evolve data protection capabilities—encryption, access control, and schema validation—across the platform.
• Build and maintain highly available, multi-tenant infrastructure for governance services.
• Establish and evangelize engineering standards, architectural patterns, and technical frameworks across the organization.
• Collaborate with executives, product management, and engineering leaders to align technical vision with business objectives.
• Mentor senior and staff engineers to grow technical leadership and foster open-source contribution.
• Contribute to relevant open-source projects and represent Confluent externally through talks and thought leadership.

Skills

Distributed systems

Leadership

Cloud-native

Schema governance

Open-source

Mentorship

Education

Master's degree

Tools

Confluent Platform

Schema Registry

Cloud (AWS/Azure/GCP)

Job description

IBM Software in San Diego is seeking a Principal Engineer to lead the architecture and delivery of large-scale governance systems for streaming data.

You will own core domains, mentor engineers, and collaborate with product leadership to balance technical depth with business impact, ensuring reliability and scale across Confluent‑like platforms.

This role requires autonomy in ambiguous spaces and a track record of shaping technology strategy across multiple teams.