Principal Machine Learning Engineer, Recommendation Platform (Feature Platform)

ByLabs; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

ByLabs is seeking a Principal Recommendation Platform Engineer in San Francisco to evolve our real-time personalization platform from rule-based to ML-powered, leading architecture and end-to-end delivery for a high-throughput system serving tens of millions of users globally.

You will own feature pipelines, a unified Feature Store, and integrations with AI agents/LLMs, driving reliability, latency, and throughput while delivering measurable business impact.

Qualifications

• 4+ years of recommender systems engineering experience at a consumer internet company.
• Proficiency in Go, Java, or C++ with clean, scalable code.
• Experience building a real-time recommendation system serving tens of millions of users.

Responsibilities

• Develop high-throughput, low-latency recommendation services and improve performance and reliability.
• Build stream-and-batch feature pipelines and a Feature Store for online/offline features.
• Identify architectural needs across scenarios and optimize workflows.
• Develop core components and drive adoption of emerging technologies including LLMs and AI agents.

Skills

Go

Java

C++

Tools

Vector databases

Key-value stores

Big-data systems

Job description

As a Principal Recommendation Platform Engineer, you will help upgrade our core recommendation architecture and feature platform, evolving our current rule-based personalization system into an industrial-grade, large-scale, real-time machine learning recommendation platform. The role spans key business scenarios, including trading products, content feeds, the community, and AI-powered investment workflows such as TradeGPT.

You will work closely with recommendation algorithm, growth product and operations, and data science teams to build a high-throughput, low-latency, highly available foundation for personalization at scale, serving more than 80 million users globally. You will lead architectural decisions, infrastructure development, and end-to-end delivery, taking the platform from zero to one in a resource-constrained, fast-moving environment.

Key Responsibilities

• Develop high-throughput, low-latency recommendation services. Continuously improve system performance and reliability, and establish robust degradation and disaster-recovery mechanisms.
• Build stream-and-batch feature computation and training data pipelines to support minute-level updates of user behavior features. Develop a Feature Store that unifies online and offline features, address temporal feature leakage and feature consistency issues, and support faster model iteration and performance improvements.
• Identify common architectural needs across recommendation scenarios—including trading products, content feeds, and community—and optimize their workflows. Build reusable platform capabilities to address technical challenges and business requirements.
• Develop and optimize core recommendation components, and drive the engineering and business adoption of emerging technologies, including LLMs and AI agents.

Major Requirements

• At least 4 years of recommender systems engineering experience at a consumer internet company. You have played a significant role in, or led, the launch or architectural modernization of a real-time recommendation system serving tens of millions of users. Experience building a recommendation system or feature platform from the ground up is preferred.
• Strong computer science fundamentals and proficiency in at least one of Go, Java, or C++. You write well-structured code, apply sound design principles, and enjoy tackling challenging engineering problems.
• Experience with stream-and-batch data processing and feature and training data pipelines, including hands-on experience resolving stream-processing latency and data backlog issues. Familiarity with the deployment and performance tuning of mainstream big-data systems, key-value stores, and vector databases.
• A platform-oriented approach to architecture and standardization. Experience developing core modules for a large-scale recommendation platform, feature platform, experimentation platform, or high-performance RPC framework is preferred.
• A deep understanding of the recommendation system lifecycle, with the ability to collaborate effectively with algorithm teams and translate model requirements into reliable production systems.

Nice-to-have:

• Architecture experience in search, advertising, or recommendations at major technology companies
• Experience as a key engineer building cross-business feature platforms, large-scale online and offline feature services, or embedding feature pipelines.
• Experience with multi-region deployment, data synchronization, active-active architectures, or cross-region RPC latency optimization.
• An AI-driven approach and hands-on experience bringing LLM or agent applications into production for search, advertising, or recommendation use cases.

Why Join Us

• A multi-domain recommendation challenge in fintech: Go beyond a single domain such as video or e-commerce. You will build personalization capabilities across trading products, market intelligence, community content, campaigns, and AI-powered investment workflows.
• Proven business impact with substantial room for ML advancement: Our current recommendation system has demonstrated a 71% increase in CTR and a 1,285% increase in conversion rate, with significant potential for an ML-native upgrade.
• Work at the intersection of AI and recommendations: Build where traditional recommender systems, generative AI, and LLM agents meet—one of the most active frontiers in engineering.
• An opportunity to lead, not just follow: Our combination of trading, content, community, AI, and campaigns creates distinctive technical challenges and enables rapid iteration.
• Global scale: Serve more than 80 million users across over 200 countries, tackling the real-world complexity of large-scale, real-time systems.