Add a simple Question Answering notebook with Haystack

[codificat]

In #9 we are exploring various QA systems.

This PR provides a simple experiment of Extractive and Generative QA using Haystack

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[codificat]

NOTE: this PR also includes the sample dataset from #11 (same commit) in order to have data to work on.

[Shreyanand]

Great start! The adversarial example was a good addition :D Haystack makes implementing the reader retriever approach quite intuitive. For the source of context, it should have a method to point back to the file, it seems to be a simple string search problem...

For the PR, I think it has the commits from the file addition PR as well. I guess if we separate them then this can be merged independently.

Looking forward to the generative example from haystack as well!

Reply via ReviewNB

[suppathak]

Line #24.            details="minimum" ## Choose from minimum, medium, and all

Looks great!! Thanks Pep.

I was observing some of the contexts. I do have some observations and suggestions in this context.

1. "All" is the choice means whole document ?
2. I also observed that some of the question got included in the context along with the answers. what if we separate the questions from the answers in the corpus text. and train our model with the text only containing the answers and not question. lmkwyt? I will also try to apply it in my QA model testing work.

Reply via ReviewNB

[codificat]

all means: show all the details about each answer, e.g. score, offset, ... vs minimum which only shows the answer and context. I thought "minimum" would be enough here, but happy to change it to show all the details.

About context that includes questions: yes, that comes from some of the source documents that do include questions. Most notably, the FAQ in the ROSA worshop: https://www.rosaworkshop.io/rosa/14-faq/

If we try to pre-process the content to separate Questions from Answers, I'm not sure how successful that would be in general because most of the answers need their respective questions in order to make sense (extreme case: there are answers that are just "Yes"). But for that document in particular (the FAQ) it might make sense to use it for the "squad-like" test... only that I have not verified if all the answers there can be obtained from other documents.

[codificat]

Converted this PR to draft while I'm working to expand it with a Generative QA approach

[codificat]

Updated with the current version that adds 3 generative QA types: RAG, LFQA and OpenAI-based.

Context now includes the full ROSA docs (plus the ROSA workshop and the MOBB material in the data/external samples)

Results are not great, I'm still trying to see if they can be improved a bit - also need to elaborate/document.

[codificat]

Ok, I believe this is ready for another review.

The RAG version is not working well for some reason that so far has escaped me. @Shreyanand @suppathak if you have suggestions especially on that part they would be most welcome.

I have added the retrieval of the whole ROSA docs from S3 storage, and these docs together with the in-repo samples (ROSA workshop and MOBB) are used for context.

There are now more comments/docs and the structure has also been updated, hopefully making it more easy to follow.

[Shreyanand]

One way to inspect the model would be to observe what retriever is fetching. Maybe it's not providing enough context to the generator. What happens if we tweak retriever model parameters?

Also. another possible explanation could be that these embedding models may not be advanced enough to capture language constructs leading to poor answers.

Reply via ReviewNB

[Shreyanand]

In essence, this notebook compares free and OS model bart_lfqa and paid Open AI model text-davinci-003 for the long form generative QA task.

For the extractive qa task it tries roberta-base-squad2 . For all of these experiments, the retriever is BM25.

Also, it has a separate experiment with combined DPR as retriever and facebook/rag-token-nq as the generator model.

Could we add this classification in the summary/conclusion? Once we have a validation dataset and metrics defined, we can add results of these experiments based on the metrics as well.

Reply via ReviewNB

[codificat]

In essence, this notebook compares free and OS model bart_lfqa and paid Open AI model text-davinci-003 for the long form generative QA task.

While it does have a free model for LFQA and an OpenAI version, the main goal of the notebook is to explore Haystack as a framework.

For all of these experiments, the retriever is BM25.

Dense vectors are used for generative QA - and a dense retriever accordingly. I expanded the introduction a bit to hopefully explain that (although I am not getting into details - should I?)

[Shreyanand]

I think the detail level is appropriate now.

While it does have a free model for LFQA and an OpenAI version, the main goal of the notebook is to explore Haystack as a framework

That makes sense.

[codificat]

Another update:

• I now removed the RAG generator test: it does not work, and deepset plan to remove the RAG generator tutorial.

• I found a problem with the Markdown pre-processor. I mention it as "fixmes" in the notebook. I am inclined NOT to try to fix these in this notebook though: the purpose of this PR is to review Haystack as a framework, and it there are issues with the markdown pre-processor they should be mentioned, right?

[Shreyanand]

/lgtm