Local RAG from Scratch using LLama3 in 5 baby steps

Try it - github.com/lancedb/vector…

1. Extract content
2. Recursive Chunking
4. Embed Chunks with LanceDB Embedding API
5. Semantic search with Query, #LLama3 for resulting output using ollama.

Simple Illustration

There are weeks where decades happen..

- Supabase Tembo - Multi-Workload Postgres Neon - Serverless Postgres go GA
- LanceDB's Lance2 unveiled
- ApacheArrow DataFusion graduates to TLP in Apache
- CrystalDB emerges from stealth

.. and what else? I feel like I forgot some stuff ..

LanceDB we’re all about having your cake and eating it too. Fast scan, random access, large blobs - other systems support at most 2 but often just 1. I call this the CAP theorem of AI data. Lance format is the only one that can do all 3 - critical for multimodal AI data

🚀Exciting News: The Lance columnar format was a game-changer in efficiently managing AI/ML workloads. But hold onto your hats because Lance v2 is here, and it's going to blow your mind! 🤯

blog.lancedb.com/lance-v2/

🗨️ Interested in building a 'Chat with PDF' application? Check out our newest tutorial featuring Chainlit LanceDB and LangChain 🗨️

Demo: buff.ly/3JlNaIl
Code: buff.ly/4cUqTPr

Sign up for Ploomber Cloud: buff.ly/3vNMfNQ

During this weeks :probabl. livestream I will work on a KNNClassifier that uses a VectorDB as a backend. In particular, I'll explore LanceDB and maybe run a benchmark or two 😄.

If that sounds fun, feel free to join here:

youtube.com/live/HIRJFYncE…

Why do we need multi-modal data formats & tensor lakes?

In our latest ep, Timothy Chen and I chat with changhiskhan about his work on LanceDB & the growing need for new data lakes that support multi-modal data to unlock the opportunity for LLM & AI.

podcasters.spotify.com/pod/show/yet-a…

🚨 New Dataset alert 🚨

Introducing the COCO-2017 dataset in LanceDB file format that allows for super-fast access during training with little CPU and memory overhead 🚀

🔧 coco2017 Lance (train) - kaggle.com/datasets/heyyt…

🔧 coco2017 Lance (val) - kaggle.com/datasets/heyyt…

⚡️Introducing Lance format for efficiently storing, loading, and processing large image datasets for machine learning and deep learning pipelines.

🚀Benefits:
- Compressed columnar storage for reduced disk space
- Fast data loading and random access
- Unified storage for images,

Track AI Trends with crewAI Agents with RAG 🔥

It works in 3 simple steps
1. Use Free News API to gather AI news
2. Make Agents and assign them tasks
3. Create RAG with LanceDB vectorDB

Colab - shorturl.at/mDJZ2
Blog - shorturl.at/euCI9

#ai #crewai #trends

Several technology trends are coming together to make a big splash for AI Infra 🧵

Let's teach LLMs to respond to instructions in Hindi 🇮🇳

How? By using the Dolly-Alpaca combined Hindi dataset with over 60K samples for instruction fine-tuning, in LanceDB's Lance file format 🚀

Check it out 🎯
kaggle.com/datasets/heyyt…

Dev - e/acc i am building on top of them like LanceDB hehe

google bard ok just saw LanceDB (which i am using) supports imagebind. this is a possible extension.

Project Launch! Multimodal Myntra Fashion Search Engine with LanceDB 🚀

I have recently been exploring Vector Databases and their use cases and thought of starting off with a simple example teaching how to get started with it.

This is the first project in the series to come

Announcing the LLM Adventures Notebook series on Kaggle, where I will be making notebooks on various interesting use-cases of LLMs and RAG pipelines using Open LLMs and datasets from Kaggle ✨

Check it out: kaggle.com/code/heyytanay…

semantic search on twitter archive tweets - semantweet search if you will, using openai small/large embeddings

also supports (thanks to sql operations in LanceDB )
- time based filtering
- link only search
- likes / rt filtering
- media only search

(p.s videos are for music)

🔥 Embedded Databases Unlocked! 🔥

5 mins is all you need to master this game-changing tech for creating RAG systems.

I broke it all down in my new blog post!

feat : LanceDB

✨ Columnar Oriented for lightning-fast analytics
✨ LANCE format for efficient compression
✨

Pratik {Coding_Geek} I have been trying out LanceDB for some time now, the simplicity of the usage appeals to me and the fact how easy it is to set up, just a pip install works?

Publishing a blog on it soon!

✨ New Lance Dataset in the town ✨

Here's the 1M tokenized sample version of the Open WebText dataset with 1.1B tokens!

Perfect for small-scale fine-tuning/knowledge distillation of Language Models powered by LanceDB file format 🚀

Check it out!
kaggle.com/datasets/heyyt…