A 100h/week job (e.g., CEO of a startup) apparently easily becomes a 24/7 job, where the job description of the remaining 68h is 'forget about what happened in those 100h, just enjoy life, family, and sleep'. And it's much much harder to succeed in those 68h than in those 100h.

Why does Chain of Thought (CoT) work?

Our #ICLR 2024 paper proves that CoT enables more *iterative* compute to solve *inherently* serial problems. Otoh, a const-depth transformer that outputs answers right away can only solve problems that allow fast parallel algorithms.

🆕 📢 voyage-large-2-instruct embedding model tops the MTEB leaderboard 🥇! huggingface.co/spaces/mteb/le…

— embedding dimension = 1024 (4x smaller than any other non-Voyage model in top-5)
— 16K context length (2x of OpenAI v3 large)

blog: blog.voyageai.com/2024/05/05/voy…

#RAG #LLM

Correction: I did the math wrong (not considering log/log scales). Sophia is ~1.6x times more efficient than Adam (thanks for pointing out Tengyu Ma).

Very cool project that not enough people talk about!

Have tried this in production, voyage ai embedding provides a few percentage of final performance improvement, despite being a small segment in the entire ML pipeline!

Bullish! 🫡

Final Update: One more magnitude of testing Sophia. We're talking model sizes in the B's, tokens in the T's. Sophia once again wins out. For me at least this is clear evidence that Sophia may be a replacement for Adam even in large scale runs.

Anthropic does not offer its own embedding model. One embeddings provider that has a wide variety of options and capabilities encompassing all four of the above considerations is Voyage AI.
Voyage AI makes state of the art embedding models and offers customized models for

⛵ Voyage AI Embedding Integration Package ↗️

Use the same custom embeddings that power Chat LangChain via the new langchain-voyageai package! Recommended by Anthropic as their preferred embedding provider, Voyage AI builds custom embedding models for your company or

Thanks for trying our optimizer! hope that Sophia can save some compute for FAIR and others :)

In increasing difficulty,

1. train artificial neural nets
2. train one’s own biological neural net
3. train others’ neural nets

Level 1.5: train others’ neural nets when others are also willing to train their own— that’s why profs can mentor even though they may fail at 2.