But wait, ChatGPT is where I go so that I don't get told to RTFM

When training LLMs, you should spend less time reading raw text and more time inspecting tokenizer outputs.

You'd be amazed how many problems creep into tokenization. For instance, the T5 tokenizer treats curly braces as <unk>.

If your model is bad at math, how is it tokenizing…

Unsloth is incredibly efficient with GPU utilization. GPU 0 is in an oven right now.

Can we all chip in for a server with 50 TB RAM so we can train a LightGBM LLM? Asking for a friend

I use Modal because it brings me joy. There isn't much more to it.

I think we all knew Llama 3 was going to be good, but this appears to be really good

I’d love to see ColBERT at 22M params (Omar Khattab Benjamin Clavié 👀)

Time-series foundational models aren't working very well because we're completely misusing the attention mechanism

Most of our recent embedding search breakthroughs were probably discovered by Google 4 years ago

SMOTE should always be your first choice for tabular data preprocessing. Class imbalance is a solved problem!

The gap between small models and huge models is closing.

Gap between the best GPT4 and 3.5T is about 140.

Gap between Opus and Haiku is only 80!

NNs are so hard to train because even in the presence of bugs, they often just work. Sure, they're limping with an undiagnosed injury, but your QKV multiplication that's actually a QQV multiplication somehow works kinda?

I've found the wrong way to use LLMs is to solve your most difficult tasks, the ones where you have expert level knowledge.

The right way is to use them to avoid wasting time on repetitive tasks so that you can focus all your brainpower on those most difficult tasks.

I don't…

What in the...