Principal Machine Learning Engineer, Recommendation Algorithm

ByLabs; San Francisco (CA); On-site; USD 180,000 - 280,000; Full time

Job summary

ByLabs is seeking a Principal Recommendation System/Platform Engineer to upgrade our core recommendation architecture, transforming a rule-based personalization system into an industrial-scale, real-time ML platform supporting trading products, content feeds, the community, and AI-powered investment workflows.

You will partner with data science, growth, and operations to deliver a high-availability foundation for personalization at scale, serving 80 million users globally.

Qualifications

• At least 4 years of recommender systems engineering experience at a consumer-facing internet company and hands-on production issues experience.
• Strong CS fundamentals and proficiency in Go, Java, or C++ with well-structured code.
• Familiarity with PyTorch or TensorFlow and end-to-end workflow of training, exporting, deploying, and serving models.

Responsibilities

• Develop high-throughput, low-latency recommendation services and ensure reliability with degradation and disaster-recovery mechanisms.
• Build and optimize large-scale model training infrastructure, incl. distributed training and job scheduling.
• Develop deployment and online inference services with autoscaling and canary releases for high traffic.
• Establish feedback loop from offline eval to production monitoring and incident diagnosis.
• Collaborate with teams to standardize training, deployment, and inference across scenarios.

Skills

Recommender systems

Go

Java

C++

PyTorch

TensorFlow

Tools

PyTorch

TensorFlow

Job description

As a Principal Recommendation System/Platform Engineer, you will lead the upgrade of our core recommendation architecture and recommendation model platform, transforming our existing rule-based personalization system into an industrial-grade, large-scale, real-time ML recommendation platform. This role spans key business scenarios, including trading products, content feeds, the community, and AI-powered investment workflows such as TradeGPT.

You will work closely with recommendation algorithm, growth product and operations, and data science teams to build a high-throughput, low-latency, highly available foundation for personalization at scale, serving more than 80 million users globally. You will lead architectural decisions, infrastructure development, and end-to-end delivery, taking the platform from zero to one in a resource-constrained, fast-moving environment.

Key Responsibilities

• Develop high-throughput, low-latency recommendation services. Continuously improve system performance and reliability, and establish robust degradation and disaster-recovery mechanisms.
• Build and optimize large-scale model training infrastructure, including distributed training, job scheduling, resource management, fault recovery, and training observability. Improve training efficiency, stability, and resource utilization.
• Build model deployment capabilities and high-performance online inference services. Optimize model loading, inference latency, throughput, and infrastructure cost; implement autoscaling, canary releases, graceful degradation, and disaster recovery to ensure service availability under high traffic.
• Establish an engineering feedback loop from offline evaluation and online experimentation to production monitoring. Monitor model quality, service performance, and anomalies; build alerting capabilities to support incident diagnosis and safe rollback.
• Collaborate with recommendation algorithm and business engineering teams to identify common needs across trading products, content feeds, community, and other recommendation scenarios. Standardize training, deployment, and inference capabilities to bring models into production reliably and efficiently.

Major Requirements

• At least 4 years of recommender systems engineering experience at a consumer-facing internet company. You have contributed substantially to the design and development of a model training platform, model deployment platform, or large-scale online inference service, and have hands‑on experience resolving production issues.
• Strong computer science fundamentals and proficiency in at least one of Go, Java, or C++. You write well-structured code, apply sound design principles, and enjoy tackling challenging engineering problems.
• Familiarity with at least one mainstream deep learning framework, such as PyTorch or TensorFlow. You understand the end-to-end workflow of training, exporting, deploying, and serving recommendation models, and have practical experience with distributed training or training job orchestration.
• An understanding of the recommendation system lifecycle and the engineering considerations involved in sequence models, multi-objective models, and other recommender model types. You can work with algorithm teams to turn model requirements into reusable, maintainable production platform capabilities.
• Experience designing and optimizing high-throughput, low-latency services. You can diagnose and resolve online inference issues involving throughput, P99 latency, reliability, and resource cost.

Nice-to-have:

• Architecture experience in search, advertising, or recommendations at major technology companies
• Experience leading the development of a training platform, model deployment platform, or online inference pipeline from the ground up in a large-scale recommendation, advertising, or search system, and making those capabilities reusable across business scenarios.
• Experience optimizing distributed training or inference for complex recommendation models, or scheduling heterogeneous CPU/GPU resources.
• Experience with cross-region high-availability deployments, active-active architectures, model release governance, or large-scale production incident response.
• An AI-driven approach and hands‑on experience bringing LLM or agent applications into production for search, advertising, or recommendation use cases.

Why Join Us

• A multi-domain recommendation challenge in fintech: Go beyond a single domain such as video or e-commerce. You will build personalization capabilities across trading products, market intelligence, community content, campaigns, and AI-powered investment workflows.
• Proven business impact with substantial room for ML advancement: Our current recommendation system has demonstrated a 71% increase in CTR and a 1,285% increase in conversion rate, with significant potential for an ML-native upgrade.
• Work at the intersection of AI and recommendations: Build where traditional recommender systems, generative AI, and LLM agents meet—one of the most active frontiers in engineering.
• An opportunity to lead, not just follow: Our combination of trading, content, community, AI, and campaigns creates distinctive technical challenges and enables rapid iteration.
• Global scale: Serve more than 80 million users across over 200 countries, tackling the real-world complexity of large-scale, real-time systems.