# RAG custom dataset

**URL:** <https://discuss.huggingface.co/t/rag-custom-dataset/1294>\
**Category:** Models\
**Created:** [September 29, 2020, 12:52pm UTC](https://discuss.huggingface.co/t/rag-custom-dataset/1294 "2020-09-29T12:52:01Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![nbroad](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nbroad/32/26465_2.png) [@nbroad](https://discuss.huggingface.co/u/nbroad)\
**Post date:** [September 29, 2020, 12:52pm UTC](https://discuss.huggingface.co/t/rag-custom-dataset/1294/1 "2020-09-29T12:52:01Z")

</div>

I just saw that Facebook AI released a [blog post about RAG](https://ai.facebook.com/blog/retrieval-augmented-generation-streamlining-the-creation-of-intelligent-natural-language-processing-models/) ( Retrieval Augmented Generation: Streamlining the creation of intelligent natural language processing models) and that it is already incorporated in the [HuggingFace API](https://huggingface.co/transformers/master/model_doc/rag.html#rag).

I looked quickly, and I couldn’t see how to use a custom dataset with it. [It seems like it will only pull down indexed datasets from HuggingFace’s AWS storage](https://github.com/huggingface/transformers/blob/0ccb6f5c6da9e703766e8053581fddfc6dcc71a9/src/transformers/configuration_rag.py#L43). I’m wondering if anyone can show me how to

1. Create an indexed dataset. I’m assuming this is just a big collection of embeddings that have been made by running documents through a model and taking the output embedding. I’m wondering which model(s) can be used, how many dimensions the embeddings are expected to be, and how to format all of these vectors.
2. Use that custom dataset with HF Rag models.

---

<div class="post-metadata">

**Author:** ![thomwolf](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thomwolf/32/2198_2.png) [@thomwolf](https://discuss.huggingface.co/u/thomwolf)\
**Post date:** [September 30, 2020, 12:31pm UTC](https://discuss.huggingface.co/t/rag-custom-dataset/1294/2 "2020-09-30T12:31:20Z")

</div>

Indeed, it’s actually very simple to do with datasets and somehow explained on this page: [https://huggingface.co/docs/datasets/faiss\_and\_ea.html](https://huggingface.co/docs/datasets/faiss_and_ea.html)

We will add an example script on this.

---

<div class="post-metadata">

**Author:** ![LeroyDyer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/leroydyer/32/26511_2.png) [@LeroyDyer](https://discuss.huggingface.co/u/LeroyDyer)\
**Post date:** [July 23, 2024, 11:30am UTC](https://discuss.huggingface.co/t/rag-custom-dataset/1294/3 "2024-07-23T11:30:17Z")

</div>

today its a dead page