Principal Software Engineer (Data Infrastructure)

Lending Club; San Francisco (CA); Hybrid; USD 180,000 - 280,000; Full time

Benefits offered by this job

Hybrid work model

Comprehensive health benefits

Generous PTO

Parental leave (up to 16 weeks)

Job summary

Lending Club seeks a Principal Data Engineer to own the architecture and platform framework that powers our downstream pipelines, data protection, and infrastructure-as-code foundations. You’ll enable teams to onboard data sources securely and productionize configurations rather than custom code.

You’ll drive scalable reusable libraries, governance, and observability, mentor engineers, and partner with Data Governance, Security, and DBAs to raise engineering standards across the data platform.

Qualifications

• Experience designing and operating large-scale data systems
• Strong understanding of data governance, security, and privacy engineering
• Proven track record delivering platform-level frameworks used by multiple teams
• Deep experience with ingestion, streaming, and CDC patterns
• Bachelor’s degree or higher in a related field or equivalent experience

Responsibilities

• Own ingestion architecture and define technical vision for a declarative ingestion framework
• Lead design and implementation of reusable platform libraries for data protection and governance
• Define infrastructure-as-code practices (Terraform, CI/CD) and safe deployment lifecycles
• Drive platform reliability, monitoring, alerting, validation, and data quality
• Mentor senior staff engineers and lead architectural decision records
• Collaborate with Data Governance, Information Security, DBAs, and data-source owners to enable scalable adoption

Skills

Leadership

Mentoring

Architectural design

Communication

Problem solving

Security & privacy

Education

Bachelor's degree or higher

Tools

Databricks

Delta Lake

Unity Catalog

Auto Loader

Terraform

CI/CD

Spark UDFs

Job description

• As a Principal Data Engineer, you’ll own the technical architecture and development of the platform frameworks and infrastructure that make downstream pipelines faster, safer, and more consistent. This includes configuration-driven ingestion frameworks, secure connectivity to source systems, reusable data protection libraries for anonymization and tokenization, and the infrastructure-as-code foundation that brings these capabilities together
• This is a framework and platform engineering role, not a pipeline-delivery role. Success means enabling teams to onboard new data sources, protect sensitive information, and promote work to production through configuration rather than custom code
• Own ingestion architecture: Define the technical vision and architecture for a declarative, configuration-driven ingestion framework that brings data from relational databases and object-store files into the raw data layer. Establish reusable patterns and contracts for downstream teams while evaluating tradeoffs between ingestion approaches
• Build scalable platform frameworks: Lead the design and implementation of reusable, versioned libraries that support policy-driven data protection, including configurable rules, governed access, consistent outputs, secure-by-default failure behavior, and audit trails. Maintain backward compatibility across shared capabilities used by multiple teams
• Set infrastructure and deployment standards: Define infrastructure-as-code practices for safe deployment, replacement, rollback, and lifecycle management, including data ownership considerations for managed pipeline objects. Establish compute, cost, and sizing guidance for large initial loads and steady-state incremental processing at billion-row scale
• Drive platform reliability and observability: Establish monitoring, alerting, reconciliation, recovery, and data validation capabilities that ensure reliable operations from source to destination. Identify and address observability gaps in underlying platform tools
• Lead evidence-based technology decisions: Conduct structured proofs of concept with measurable success criteria, document architectural decisions and tradeoffs, and engage technology vendors to resolve product gaps and influence future capabilities
• Drive cross-functional platform delivery: Translate ambiguous technical challenges into clear requirements, interfaces, dependencies, and phased delivery plans. Partner with Data Engineering, Data Governance, Information Security, DBAs, and source-system owners to deliver early solutions while building scalable long-term capabilities and driving framework adoption
• Raise engineering standards: Mentor senior and staff engineers, lead technical design and code reviews, and establish best practices for testing, documentation, release management, and maintainability across shared platform capabilities

Benefits

• Untracked vacation for salaried employees, which means take it when you need it, and generous paid time off for hourly team members
• A hybrid work model that balances in-office and work from home
• Leading health and safety protocols related to COVID, including complimentary at-home testing kits from Cue Health for those coming to the office
• Up to 16 weeks paid leave for new parents, plus a fully paid, phased return-to-work policy for qualified leaves
• Mothers Rooms and hospital-grade pumps in every LC office
• Generous and varied mental health benefits
• Wellness program with cash incentives — earn up to $75 per month for being active
• Paid volunteer time and donation matching
• Onsite gym, locker room, bike room, and fitness classes (depending on location)
• Café with our own in-house barista
• Stocked pantries with healthy snacks and drinks
• Technology-forward and collaborative office spaces

You have deep experience applying AI tools to complex engineering work, including architecture exploration, development, testing, and problem-solving. You help shape how AI is incorporated into engineering workflows while maintaining strong judgment around security, accuracy, maintainability, and where human oversight remains essentialExcellent written and verbal communication skills, including the ability to develop clear technical designs, architecture decision records, interface contracts, and documentation while influencing senior stakeholders without direct authorityExperience designing and operating large-scale data systems processing billions of rows, including performance testing, cost optimization, monitoring, and recovery strategiesYou navigate ambiguous technical challenges, evaluate tradeoffs thoughtfully, and take ownership of delivering durable solutions that improve engineering effectiveness across teamsDeep hands-on expertise with Databricks, including Delta Lake, Unity Catalog, Auto Loader, workflows, compute configuration, and governance capabilities, or equivalent data platform experienceBachelor’s degree or higher, or equivalent combination of education and experienceDeep experience building ingestion capabilities across diverse data sources, including relational databases, files, APIs, and custom systems, with expertise in incremental processing, change data capture (CDC), schema evolution, and protecting production source systems from excessive loadStrong experience with infrastructure as code, including Terraform and CI/CD pipelines for data platforms10+ years of software or data platform engineering experience, including designing and delivering platform-level frameworks, libraries, or infrastructure used across multiple engineering teamsStrong knowledge of security and privacy engineering for sensitive data, including encryption, key management, tokenization or format-preserving techniques, role-based access controls, and audit loggingAdvanced programming skills in Python, SQL, and a JVM language such as Scala or Java, with experience developing Spark UDFs and reusable libraries using strong testing, packaging, and versioning practicesExperience in banking, lending, or another regulated environment with model risk management, SOX, or similar audit requirementsExperience with data quality, lineage, catalog tooling, and data contract approachesPrior experience as a Principal Engineer, Staff Engineer, or technical lead for a platform engineering teamExperience with Databricks Asset Bundles or equivalent deployment tooling