Yi Cao

Meta Superintelligence Lab (2024/05-Present)

I work on LLM measurement projects:

• Designed the first version of the Meta AI RAG failure taxonomy for the team.
• Built LLM evaluation judges that the modeling team can iterate on to improve RAG and LLM response quality.
• The work I drove not only changed the team's trajectory, but also directly unblocked the product launch (Top 2% in review).

Marketplace (Facebook platform to help people find commerce products) (2020/06-2024/05)

I worked on the search team:

• Proactively identified opportunities to improve search ranking models by understanding underperformed query segments and adding user signals. My work helped the team achieve an on-par relevance score (measured by NDCG) with competitors.
• Reviewed A/B tests and maintained a high bar for launch.
• Set goals for the team and communicated the team's progress to the leads.

I worked on the recommendation / personalization team (Python, HIVE, SQL, Airflow):

• Predicted customers' next browsing page and pre-cached it to improve the onsite experience by reducing the page load time from 1,500 ms to 50 ms.
• Applied SGT (Sequence Graph Transformation) to obtain customer browsing history embeddings, and applied clustering analysis to understand customer behavior.
• Built an XGB learning-to-rank model to predict customers' most interested products. The model was served on the homepage to make recommendations, and it also achieved a 10% CTR lift in the A/B test, which amounted to approximately $2 million in incremental revenue yearly.
• Implemented machine learning models in Airflow.
• Collaborated with analysts, engineers, and product managers for model launch and post-test analysis.
• Conducted interviews and led intern projects.

Translated business requirements into analytics solutions by leveraging various methods: ad-hoc analysis, predictive modeling, hypothesis testing, etc.

I worked on consumer analytics and marketing projects:

• Built predictive models to analyze customer retention and call volume. Identified actionable opportunities to reduce pain points in the customer journey, and made suggestions to the business.
• Built an uplift model to analyze marketing campaign results.
• Built a Tweedie XGB model to predict customer lifetime value to better target customers through direct mail.
• Conducted a cost-benefit analysis for model results and made recommendations to the business.