examples/tutorials/example_data_augmentation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Data Augmentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we provide an example of how to perform data augmentation on text. Data augmentation is to make copies and add meaningful changes to the original text data. This method can hence increase the amount of the data and provides another way to have more training data. The hope and idea behind data augmentation is that the increased amount of data can increase performance when training a model. \n",
    "\n",
    "In this example we will test if augmentation on ~1000 tweets can improve the performance of a sentiment model. I had expected an increase in performance, but such increase was not revealed by the experiment. However, the method  might be succesful on other types of text data. Also, I could imagine that data augmentation will prove useful when the Danish ressources for synonyms and word embeddings are more developed and mature, as data augmentation seems promising on English text data (see for instance the paper by Wei & Zou (2019): \"Eda: Easy data augmentation techniques for boosting performance on text classification tasks\" and the paper by Sun & He (2020): \"A novel approach to generate a large scale of supervised data for short text sentiment analysis\").\n",
    "\n",
    "Another small experiment that is carried out at the end of this tutorial is to test if we by creating tricky sentences including negations can bias the model to better handled ambiguous sentences. \n",
    "\n",
    "Steps in this tutorial:\n",
    "1. Load libraries, data etc.\n",
    "2. Preprocess the data (cleaning)\n",
    "3. Augment the data: Add spelling mistakes, swap words with word embeddings and/or synonyms\n",
    "4. Prepare the data for training and evaluation with SpaCy\n",
    "5. Train two models: with and without the augmented data\n",
    "6. Test the performance of the better performing model\n",
    "7. Test negations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Load libraries, data etc. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Before you start the tutorial, you need to install some packages. You install them by downloading the file \"requirements_dataaugmentation.txt\" and in your terminal type the command 'pip install -r requirements_dataaugmentation.txt'. \n",
    "\n",
    "#### Also, you need to install the latest version of the danlp package, which you do by typing the following command in the terminal: 'pip install git+https://github.com/alexandrainst/danlp.git'\n",
    "\n",
    "After that, you are ready to import the libraries. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import operator\n",
    "import numpy as np\n",
    "import re\n",
    "import random\n",
    "from random import shuffle\n",
    "random.seed(1)\n",
    "import os\n",
    "import srsly\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "import danlp\n",
    "from danlp.download import DEFAULT_CACHE_DIR\n",
    "from danlp import utils\n",
    "from danlp.models import load_spacy_model\n",
    "from danlp.models.embeddings  import load_wv_with_gensim\n",
    "from danlp.datasets import DanNet\n",
    "from spacy.gold import docs_to_json\n",
    "import spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the spaCy model \n",
    "nlp = load_spacy_model()\n",
    "\n",
    "# Load the word embeddings to use\n",
    "word_embeddings = load_wv_with_gensim('conll17.da.wv')\n",
    "\n",
    "# Load the synonyms to use \n",
    "dannet = DanNet()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset used in this tutorial is The Twitter sentiment, which is a small manually annotated dataset by the Alexandra Institute. It contains tags in two sentiment dimension: analytic: ['subjective' , 'objective'] and polarity: ['positive', 'neutral', 'negative' ]. In this tutorial we will only use the dimension of polarity. The data is split in train and test part. \n",
    "\n",
    "\n",
    "#### Due to Twitters privacy policy, people are allowed to delete their tweets. Therefore, to download the actual tweet text one need a Twitter development account and to generate the sets of login keys, read how to get started here: https://python-twitter.readthedocs.io/en/latest/getting_started.html. \n",
    "#### Then the dataset can be loaded with the DaNLP package (see code block below) by setting the following environment variable for the keys you get when you create an account:\n",
    "\n",
    "TWITTER_CONSUMER_KEY, TWITTER_CONSUMER_SECRET, TWITTER_ACCESS_TOKEN, TWITTER_ACCESS_SECRET"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from danlp.datasets import TwitterSent\n",
    "twitSent = TwitterSent()\n",
    "\n",
    "df_test, df_train_all = twitSent.load_with_pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### If you want to use your own data, you can also do that. Just make sure you have two columns called 'text' and 'polarity'. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We select only the two relevant columns of the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train_all = df_train_all[['text', 'polarity']]\n",
    "df_test = df_test[['text', 'polarity']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We split the training data in training and evaluation data (80/20) in order to avoid that we evaluate and test on the same data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train = df_train_all.sample(frac=0.8,random_state=200)\n",
    "df_dev = df_train_all.drop(df_train.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Preprocess data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We make all letters lowercase, remove all punctuation in order to identify as many word embeddings and synonyms later in the process. Also, we remove hashtags and links to webpages. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cleantweets(tweet):\n",
    "    clean_tweets = []\n",
    "    for i in tweet:\n",
    "        i = i.lower()\n",
    "        i = re.sub(r'[\\.\\,\\\"\\!\\?\\:\\;\\_\\=\\(\\)\\|\\*\\@\\&\\$\\\"\\’\\/\\%\\+]+',\"\",i)\n",
    "        clean_tweets.append(' '.join(re.sub(\"(@[A-Za-z0-9]+)|(#[A-Za-z0-9]+)|(@ [A-Za-z0-9]+)|(# [A-Za-z0-9]+)|(\\w+:\\/\\/\\S+)\",\" \",i).split()))\n",
    "    return clean_tweets\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train['text'] = cleantweets(df_train['text'])\n",
    "df_dev['text'] = cleantweets(df_dev['text'])\n",
    "df_test['text'] = cleantweets(df_test['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 Augment the data: Add spelling mistakes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### This section provides three different ways to add spelling mistakes to the data:\n",
    "1. To randomly delete words \n",
    "2. To randomly delete characters\n",
    "3. To randomly swap characters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. takes as input the sentence and the percentage of words to be deleted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def random_deletion_word(sentence, p_w):\n",
    "    words = [word for word in sentence.split(' ') if word is not '']\n",
    "    if len(words) == 1: # if there is only one word in the sentence, we will not delete any words\n",
    "        return words\n",
    "    \n",
    "    new_sentence = [word for word in words if random.uniform(0, 1) > p_w] # remove words\n",
    "    \n",
    "    if len(new_sentence) == 0: # if there is no words left, we want to return a random word from the sentence\n",
    "        return random.choice(words)\n",
    "    new_sentence = ' '.join(new_sentence)\n",
    "    return new_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Denne funktion fjerner ord tilfældigt med den procentsats der defineret i p_w'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_deletion_word('Denne funktion fjerner ord tilfældigt med den procentsats der er defineret i p_w', p_w=0.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. takes as input the sentence and the percentage of characters to be deleted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def random_deletion_character(sentence, p_c_d):\n",
    "    chars = [char for char in sentence]\n",
    "    if len(chars) == 1: # if there is only one char in the sentence, we will not delete any chars\n",
    "        return chars\n",
    "    \n",
    "    new_sentence = [char for char in chars if random.uniform(0, 1) > p_c_d] # remove characters\n",
    "    \n",
    "    if len(new_sentence) == 0: # if there is no characters left, we want to return a random char from the sentence\n",
    "        return random.choice(chars)\n",
    "    new_sentence = ''.join(new_sentence)     \n",
    "    return new_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Dnne funktion fjerner ogstaver tilfældigt med den procetsats der er defineet ip_c_d'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_deletion_character('Denne funktion fjerner bogstaver tilfældigt med den procentsats der er defineret i p_c_d', p_c_d=0.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. takes as input the sentence and the percentage of characters to be swapped"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def character_replacement(sentence, p_c_r):\n",
    "    chars = [char for char in sentence]\n",
    "    new_sentence = []\n",
    "    \n",
    "    for ch in chars:\n",
    "        new_char = list('qwertyuiopåasdfghjklæøzxcvbnm')\n",
    "        r = random.uniform(0, 1)\n",
    "        if r > p_c_r:\n",
    "            new_sentence += ch\n",
    "        else:\n",
    "            new_sentence += random.choice(new_char)\n",
    "            \n",
    "    new_sentence = ''.join(new_sentence)       \n",
    "    return new_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Denne funktiqn udskifter bogstaver med denyprocentsais deq er defineret i p_c_r'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "character_replacement('Denne funktion udskifter bogstaver med den procentsats der er defineret i p_c_r', p_c_r=0.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 Augment the data: Swap words with word embeddings and/or synonyms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### This function pos_extraction() takes a sentence as input and returns a list of the words and the respective pos-tags. \n",
    "\n",
    "We want to be able to swap words, but we also want the new text to be meaningful and grammatically correct. The tracking of pos-tags will be used to secure, in the final function for augmenting sentences, that only words with the defined pos-tags will be swapped and also that the words will only be swapped with a synonym/word embedding having the same pos-tag.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pos_extraction(sentence):\n",
    "    doc = nlp(sentence)\n",
    "    words = [token.text for token in doc]\n",
    "    pos = [token.pos_ for token in doc]\n",
    "    words_pos = [[i,j] for i, j in zip(pos, words)] \n",
    "    return words_pos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['DET', 'Denne'],\n",
       " ['NOUN', 'funktion'],\n",
       " ['VERB', 'returnerer'],\n",
       " ['NOUN', 'sætningens'],\n",
       " ['NOUN', 'pos-tags'],\n",
       " ['CCONJ', 'eller'],\n",
       " ['NOUN', 'ordklasser'],\n",
       " ['ADP', 'som'],\n",
       " ['PRON', 'de'],\n",
       " ['ADV', 'også'],\n",
       " ['VERB', 'kaldes']]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pos_extraction('Denne funktion returnerer sætningens pos-tags eller ordklasser som de også kaldes')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### This function get_embedding() takes a word as an input and returns a list of word embeddings to the word. \n",
    "\n",
    "It has a boundary in that it only returns the word embeddings which have a similarity score above 0.80 to the original word. This is to ensure meaningful swaps. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_embedding(word):\n",
    "    new_embeddings = []\n",
    "    try:\n",
    "        new_embeddings = [em[0] for em in word_embeddings.most_similar(word) if em[1] > 0.80] \n",
    "        if word in new_embeddings:\n",
    "            new_embeddings.remove(word)\n",
    "    except Exception as e:\n",
    "        pass\n",
    "    return new_embeddings\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['partiformand',\n",
       " 'partileder',\n",
       " 'økonomiminister',\n",
       " 'fogh',\n",
       " 'thorning-',\n",
       " 'finansminister',\n",
       " 'thorning-schmidt',\n",
       " 'udenrigsminister',\n",
       " 'regeringsleder',\n",
       " 'statminister']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_embedding('statsminister')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### This function embedding_replacement() takes a sentence, the number of words to swap (n_embed) and up to three pos tags (pos1, pos2, pos3) as input, and returns the augmented sentence. \n",
    "\n",
    "Please note, that even though you set n_embed to 5 or even 10, it might not swap that many words. It swaps up until 5 or 10, but can only swap as many words as it finds embeddings for. Most of the time the swaps are meaningful, but cases will appear where the swap makes no sense or where the embedding have the opposite meaning of the word it is swapped with.\n",
    "\n",
    "You can find the use of pos_extraction() and get_embedding() in this function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def embedding_replacement(sentence, n_embed, pos1,pos2=None,pos3=None):\n",
    "    pos_tag = [i[0] for i in pos_extraction(sentence) if i[0] in {pos1,pos2,pos3}] \n",
    "    word = [i[1] for i in pos_extraction(sentence) if i[0] in {pos1,pos2,pos3}]\n",
    "    words_to_swap = [[i,j] for i, j in zip(pos_tag, word)] # extract relevant pos-tag and word and combine   \n",
    "    \n",
    "    new_sentence = [word for word in sentence.split(' ') if word is not ''] # prepare for swapping\n",
    "    num_replaced = 0\n",
    "    for pos in random.sample(words_to_swap, len(words_to_swap)):\n",
    "        embeddings = get_embedding(pos[1])\n",
    "        embed_pos_list = pos_extraction(' '.join(embeddings)) # get embeddings and the pos-tag for embeddings\n",
    "        embed_to_be_used = [em[1] for em in embed_pos_list if em[0] == pos[0]]          \n",
    "        if len(embed_to_be_used) >= 1:  # to swap n_embed times          \n",
    "            new_sentence = [random.choice(embed_to_be_used) if word == pos[1] else word for word in new_sentence]\n",
    "            num_replaced += 1   \n",
    "        if num_replaced >= n_embed:\n",
    "            break\n",
    "    new_sentence = ' '.join(new_sentence)\n",
    "    return new_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'det er sjovest at eksperimentere med sproget og skifte sætningen ud'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_replacement('det er sjovt at eksperimentere med sproget og skifte ordene ud', n_embed=5, pos1='NOUN',pos2='ADJ',pos3='ADV')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### This function synonym_replacement() takes a sentence, the number of words to swap (n_syno) and up to three pos tags (pos1, pos2, pos3) as input, and returns the augmented sentence. \n",
    "\n",
    "Please note, that even though you set n_syno to 5 or even 10, it might not swap that many words. It swaps up until 5 or 10, but can only swap as many words as it finds synonyms for. Most of the time the swaps are meaningful, but cases will appear where the swap makes no sense due to words that can have two pos-tags dependent on the context. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def synonym_replacement(sentence, n_syno, pos1,pos2=None,pos3=None):\n",
    "    pos_tag = [i[0] for i in pos_extraction(sentence) if i[0] in {pos1,pos2,pos3}] \n",
    "    word = [i[1] for i in pos_extraction(sentence) if i[0] in {pos1,pos2,pos3}]\n",
    "    words_to_swap = [[i,j] for i, j in zip(pos_tag, word)] # extract relevant pos-tag and word and combine \n",
    "    \n",
    "    new_sentence = [word for word in sentence.split(' ') if word is not ''] # prepare for swapping\n",
    "    num_replaced = 0\n",
    "    for pos in random.sample(words_to_swap, len(words_to_swap)):\n",
    "        syno_pos_list = pos_extraction(' '.join(dannet.synonyms(pos[1]))) # get synos and the pos-tag for synos\n",
    "        syno_to_be_used = [syn[1] for syn in syno_pos_list if syn[0] == pos[0]] \n",
    "        if len(syno_to_be_used) >= 1:\n",
    "            new_sentence = [random.choice(syno_to_be_used) if word == pos[1] else word for word in new_sentence]\n",
    "            num_replaced += 1   \n",
    "        if num_replaced >= n_syno:\n",
    "            break\n",
    "    new_sentence = ' '.join(new_sentence)\n",
    "    return new_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Det er en vanskelig funktion at arbejde med sprogbrug og teknologi'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "synonym_replacement('Det er en svær opgave at arbejde med sprog og teknologi', n_syno=5, pos1='NOUN',pos2='ADJ',pos3='ADV')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3 Augment the data: Putting it all together"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### This function takes a dataframe with columns 'text' and 'polarity' as input, as well as three pos-tags to swap words from. Next it takes the number of embeddings and number of synonyms to swap. Lastly, the number of sentences to augment, as well as the percentage of words to be deleted and the percentage of characters to be respectively deleted and swapped are taken as inputs. It returns a new dataframe with both the original sentences and the augmented sentences. \n",
    "\n",
    "You have to specify at least one pos-tag, but the other two are optional. You can select all or only some of the augmentation methods by specifying the arguments. If nothing is specified the default is 0 to all optional arguments, and the respective augmentation will not be done. Be aware that swapping embedding is very time-consuming.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "def augment_sentences(df,pos1,pos2=None,pos3=None, n_embed=0, n_syno=0, p_w=0, p_c_d=0, p_c_r=0, n_sen=1):\n",
    "    augmented_sentences=[]\n",
    "    polarity=[]\n",
    "    for i in df['text']:\n",
    "        augmented_sentences.append(i)\n",
    "        for _ in range (n_sen):\n",
    "            if n_embed >= 1:\n",
    "                i = embedding_replacement(i,n_embed,pos1,pos2,pos3)\n",
    "            if n_syno >= 1:\n",
    "                i = synonym_replacement(i,n_syno,pos1,pos2,pos3)\n",
    "            if p_w > 0:\n",
    "                i = random_deletion_word(i, p_w)\n",
    "            if p_c_d > 0:\n",
    "                i = random_deletion_character(i, p_c_d)\n",
    "            if p_c_r > 0:\n",
    "                i = character_replacement(i, p_c_r)\n",
    "            augmented_sentences.append(i)\n",
    "        \n",
    "    for j in df['polarity']:\n",
    "        for _ in range (n_sen+1):\n",
    "            polarity.append(j)    \n",
    "    total_df = pd.DataFrame({'text':augmented_sentences,'polarity':polarity})\n",
    "    return total_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we swap both synonyms and embeddings as well as adding small spelling mistakes to the same sentence. This is done to achieve the most variation so that the augmented sentences will not be too alike to the original sentence and hopefully have a better chance of improving the model. \n",
    "\n",
    "We try with two different combinations of pos-tags because the Danish ressources are limited and we aim to achieve as much variation as possible in the augmented sentences. \n",
    "\n",
    "In total we end up with 8 new sentences out of each sentence. This number is chosen based on the recommendations from Wei & Zou (2019) in their paper: \"Eda: Easy data augmentation techniques for boosting performance on text classification tasks\" as well as taking into account that the Danish ressources are less developed than English ressources. Therefore we have 8 augmented sentence instead of the 12 that their recommendations would be with ~1000 sentences. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_noun_adj_adv = augment_sentences(df_train, pos1='NOUN',pos2='ADJ',pos3='ADV', n_embed=5, n_syno=5, p_w=0.01, p_c_d=0.005, p_c_r=0.005,n_sen=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_verb_propn_adv = augment_sentences(df_train, pos1='VERB',pos2='PROPN',pos3='ADV', n_embed=5, n_syno=5, p_w=0.01, p_c_d=0.005, p_c_r=0.005,n_sen=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#combine the two created dataframes\n",
    "df_ori_aug = df_noun_adj_adv.append(df_verb_propn_adv, ignore_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the function keeps the original sentence, we now have it twice, therefore we drop the dublicates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ori_aug.sort_values(\"text\", inplace = True) \n",
    "df_ori_aug.drop_duplicates(subset =\"text\", \n",
    "                     keep = 'first', inplace = True) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Prepare the data for training and evaluation with SpaCy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### In this section, we prepare the data for training two SpaCy models on sentiment. One model with only the original data (df_train) and one with the original as well as the augmented data (df_ori_aug). \n",
    "\n",
    "To do so, we change the labels of polarity and the prepare_data function converts it to json format. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train['polarity'] = df_train['polarity'].replace(['positiv','negativ','neutral'],[0,2,1])\n",
    "df_ori_aug['polarity'] = df_ori_aug['polarity'].replace(['positiv','negativ','neutral'],[0,2,1])\n",
    "df_dev['polarity'] = df_dev['polarity'].replace(['positiv','negativ','neutral'],[0,2,1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp_prep = load_spacy_model()\n",
    "nlp_prep.disable_pipes(*nlp_prep.pipe_names)\n",
    "sentencizer = nlp_prep.create_pipe(\"sentencizer\")\n",
    "nlp_prep.add_pipe(sentencizer, first=True)\n",
    "# function to read pandas dataFrame and save as json format expected by spaCy\n",
    "def prepare_data(df, outputfile):\n",
    "    # choose the name of the columns containg the text and labels\n",
    "    label='polarity'\n",
    "    text = 'text'\n",
    "    def put_cat(x):\n",
    "        # adapt the name and amount of labels\n",
    "        return {'positiv': bool(x==0), 'neutral': bool(x==1), 'negativ': bool(x==2)} \n",
    "    \n",
    "    cat = list(df[label].map(put_cat))\n",
    "    texts, cats= (list(df[text]), cat)\n",
    "    \n",
    "    #Create the container doc object\n",
    "    docs = []\n",
    "    for i, doc in enumerate(nlp_prep.pipe(texts)):\n",
    "        doc.cats = cats[i]\n",
    "        docs.append(doc)\n",
    "    # write the data to json file\n",
    "    srsly.write_json(outputfile,[docs_to_json(docs)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prepare_data(df_train, 'ori.json')\n",
    "prepare_data(df_ori_aug, 'syn_ori_aug.json')\n",
    "prepare_data(df_dev, 'dev.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Train two models: with and without the augmented data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We are now ready to train the models. You need to open your terminal (and activate your virtual environment if you are working in one) and go to the folder where this tutorial is placed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You need to find the directory (DEFAULT_CACHE_DIR) of the spacy model and word embeddings that you will train on. \n",
    "from danlp.download import DEFAULT_CACHE_DIR\n",
    "DEFAULT_CACHE_DIR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Change 'DEFAULT_CACHE_DIR' in the following two commands to what is printed by the last command. In the command it is defined that a Danish model should be trained (da), what the training data and evaluation data is, that it is a categorical model (textcat) and which vectors to use, as well as how many iterations to run. \n",
    "\n",
    "\n",
    "#### When you have done that, copy the commands by turn into the terminal and run the training. "
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "python3 -m spacy train da model_ori ori.json dev.json --pipeline textcat --vectors 'DEFAULT_CACHE_DIR/wiki.da.wv.spacy' --n-iter 10 "
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "python3 -m spacy train da model_ori_aug ori_aug.json dev.json --pipeline textcat --vectors 'DEFAULT_CACHE_DIR/wiki.da.wv.spacy' --n-iter 10 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### From the output in the terminal, you can see which of the 10 models performs the best. \n",
    "\n",
    "We want to choose the one with the highest F1-score and lowest loss. The better model for each model is chosen:\n",
    "\n",
    "model_ori: model5         loss: 0.114   f1_score: 53.684\n",
    "\n",
    "model_ori_aug: model5     loss: 0.002   f1_score: 50.829\n",
    "\n",
    "#### In this example, the slightly better model is model5 with only the original data, therefore we will further test the performance of this model. \n",
    "You might get different results depending on your choices in the augmentation proces."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Test the performance of the better performing model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We load the model, and we use the model in the functions in order to predict the sentiment and held the prediction against the true sentiment.  \n",
    "\n",
    "Thereafter we evaluate by a classification report with individual precision, recall and f1-score for each label of sentiment as well as a macro average of each measure.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py:193: UserWarning: [W019] Changing vectors name from da_model.vectors to da_model.vectors_312956, to avoid clash with previously loaded vectors. See Issue #3853.\n",
      "  \"__main__\", mod_spec)\n"
     ]
    }
   ],
   "source": [
    "#load the trained model\n",
    "output_dir ='model_ori/model5'\n",
    "new_nlp = spacy.load(output_dir)\n",
    "\n",
    "def predict(x):\n",
    "    doc = new_nlp(x)\n",
    "    return max(doc.cats.items(), key=operator.itemgetter(1))[0]\n",
    "\n",
    "result = pd.DataFrame({'pred':[predict(i) for i in df_test['text']],'true':df_test['polarity'],'text':df_test['text']})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "     negativ       0.64      0.87      0.74       271\n",
      "     neutral       0.48      0.19      0.27       120\n",
      "     positiv       0.53      0.40      0.46       121\n",
      "\n",
      "    accuracy                           0.60       512\n",
      "   macro avg       0.55      0.49      0.49       512\n",
      "weighted avg       0.58      0.60      0.56       512\n",
      "\n"
     ]
    }
   ],
   "source": [
    "result['true'] = result['true'].replace([0,2,1],['positiv','negativ','neutral'])\n",
    "print(classification_report(result['true'],result['pred']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems that the model is better with negative sentiment compared to especially neutral sentiment, where the positive sentiment places itself in the middle."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Test negations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Usually the model have a hard time and fails to predict the sentiment on sentences with negations. Therefore, in this section we will test if we can unbias the model's predictions on sentences with negations by exposing the model to many different examples. \n",
    "\n",
    "We have chosen the better performing model from the previous research and will now compare it to a model trained on the same data + created negation sentences that are both positive and negative. \n",
    "\n",
    "When both have been trained, we will test their performance on ambigious sentences with negations but also on the test dataset we used to test the better model on previously. This way we will explore if the extra sentences with negations can improve the performance of the model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Making lists to create different sentences\n",
    "adj_pos=['glad','fornøjet','dejlig','tilpas','demokratisk','dygtig','flot','grundig','klog','menneskelig','opmærksom','sød','venlig','imødekommende']\n",
    "adj_neg = ['sur','irriteret','utilfreds','negativ','frustreret','vred','ucharmerende','voldsom','utilgængelig','utilnærmelig']\n",
    "adj_beskrivende = ['skøn','fremragende','dejlig','behagelig','underholdende']\n",
    "\n",
    "adv = ['rigtig','meget','lidt','noget','virkelig','umådelig','rimelig','forholdsvis','temmelig']\n",
    "\n",
    "noun_person = ['manden', 'drengen','damen','Hans','Ulla','Lotte','Carsten','han','hun','pigen',]\n",
    "noun_profession = ['student','læge','pædogog','håndværker','bygmester','arkitekt','politiker','datalog','mor','far','musiker','underviser']\n",
    "noun_ting = ['biograftur', 'middag','sejltur','oplevelse','debat']\n",
    "noun_ting2 = ['maden','selskabet','servicen','stemningen','personalet']\n",
    "\n",
    "verb_neg = ['afskyr','hader','foragter','ringeagter']\n",
    "verb_pos = ['danse','synge','debattere','arbejde','spise','snakke']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Negations for training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# looping through different combinations of sentences\n",
    "negations = []\n",
    "label = []\n",
    "words= []\n",
    "l = [adj_pos, adj_neg, adj_beskrivende, adv, noun_person, noun_profession, noun_ting, noun_ting2,verb_neg, verb_pos]\n",
    "sen_pos = ['{} er ikke {} i dag','{} er {} i dag','{} er en {} {}','Det har været en {} {}','Jeg vil {} mere','Hvem vil ikke gerne være en {} {}?','Der er ingen grund til at være {} når {} er så godt']\n",
    "sen_neg = ['{} er ikke {} i dag','{} er {} i dag','{}! Men ellers tak','Hvis du er {}, kan du ikke være med','Jeg vil aldrig {} mere','hvem vil {}? Absolut ikke mig!', 'Jeg {} folk der ikke viser respekt']\n",
    "for i in range(10):\n",
    "    w = [random.choice(x) for x in l]\n",
    "    negations.append(sen_pos[0].format(w[4],w[1]))\n",
    "    negations.append(sen_pos[1].format(w[4],w[0]))\n",
    "    negations.append(sen_pos[2].format(w[4],w[0],w[5]))\n",
    "    negations.append(sen_pos[3].format(w[2],w[6]))\n",
    "    negations.append(sen_pos[4].format(w[9]))\n",
    "    negations.append(sen_pos[5].format(w[0],w[5]))\n",
    "    negations.append(sen_pos[6].format(w[1],w[7]))   \n",
    "    label += len(sen_pos) * [0] #positiv\n",
    "    negations.append(sen_neg[0].format(w[4],w[0]))\n",
    "    negations.append(sen_neg[1].format(w[4],w[1]))\n",
    "    negations.append(sen_neg[2].format(w[2]))\n",
    "    negations.append(sen_neg[3].format(w[1]))\n",
    "    negations.append(sen_neg[4].format(w[9]))\n",
    "    negations.append(sen_neg[5].format(w[9]))\n",
    "    negations.append(sen_neg[6].format(w[8]))\n",
    "    label += len(sen_pos) * [2] #negativ\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# saving as a dataframe\n",
    "negations_train = pd.DataFrame({'text':negations,'polarity':label})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Negations for testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "negations = []\n",
    "label = []\n",
    "l = [adj_pos, adj_neg, adj_beskrivende, adv, noun_person, noun_profession, noun_ting,noun_ting2,verb_neg, verb_pos]\n",
    "sen_pos = ['{} er {} {}','Den {} gjorde at en {} begyndte at {} meget','Efter en gåtur var {} i en {} stemning','En {} {} kan løfte humøret på selv en ikke så lidt {} gammel mand']\n",
    "sen_neg = ['{} er ikke {} {}','Den {} gjorde at en {} ikke gad at {}','Det har været en {} {}. Men med det sagt, {} jeg virkelig {}','Jeg vil ikke have hjælp fra {}']\n",
    "for i in range(10):\n",
    "    w = [random.choice(x) for x in l]\n",
    "    negations.append(sen_pos[0].format(w[7],w[3],w[2]))\n",
    "    negations.append(sen_pos[1].format(w[6],w[5],w[9]))\n",
    "    negations.append(sen_pos[2].format(w[4],w[2]))\n",
    "    negations.append(sen_pos[3].format(w[0],w[5],w[1]))\n",
    "    label += len(sen_pos) * [0] #positiv\n",
    "    negations.append(sen_neg[0].format(w[7],w[3],w[2]))\n",
    "    negations.append(sen_neg[1].format(w[6],w[5],w[9]))\n",
    "    negations.append(sen_neg[2].format(w[2],w[6],w[8],w[7]))\n",
    "    negations.append(sen_neg[3].format(w[4]))\n",
    "    label += len(sen_neg) * [2] #negativ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# saving as a dataframe\n",
    "negations_test = pd.DataFrame({'text':negations,'polarity':label})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We append the negations to the dataset that returned the better performance. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ori_negations = df_train.append(negations_train, ignore_index = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We load the model and the function transforms the data to json format. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the spaCy model with the adjusted pipeline for the preparation of data to work\n",
    "nlp_prep = load_spacy_model()\n",
    "nlp_prep.disable_pipes(*nlp_prep.pipe_names)\n",
    "sentencizer = nlp_prep.create_pipe(\"sentencizer\")\n",
    "nlp_prep.add_pipe(sentencizer, first=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prepare_data(df_ori_negations, 'ori_negations.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### As before, open the terminal and type in the following (remember to change 'DEFAULT_CACHE_DIR):"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "python3 -m spacy train da model_ori_neg ori_negations.json dev.json --pipeline textcat --vectors 'DEFAULT_CACHE_DIR/wiki.da.wv.spacy' --n-iter 10 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The better model here is model8 with loss: 0.032   f1_score: 48.438"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Now we can load both models and test the performance of the models on the created negation sentences for test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py:193: UserWarning: [W019] Changing vectors name from da_model.vectors to da_model.vectors_312956, to avoid clash with previously loaded vectors. See Issue #3853.\n",
      "  \"__main__\", mod_spec)\n",
      "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py:193: UserWarning: [W019] Changing vectors name from da_model.vectors to da_model.vectors_312956, to avoid clash with previously loaded vectors. See Issue #3853.\n",
      "  \"__main__\", mod_spec)\n"
     ]
    }
   ],
   "source": [
    "#load the models\n",
    "output_dir1 ='model_ori/model5'\n",
    "nlp = spacy.load(output_dir1)\n",
    "\n",
    "output_dir2 ='model_ori_neg/model8'\n",
    "nlp_negation = spacy.load(output_dir2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "#use the models for predictions\n",
    "def predict(x):\n",
    "    doc = nlp(x)\n",
    "    return max(doc.cats.items(), key=operator.itemgetter(1))[0]\n",
    "\n",
    "def predict_negation(x):\n",
    "    doc = nlp_negation(x)\n",
    "    return max(doc.cats.items(), key=operator.itemgetter(1))[0]\n",
    "\n",
    "result_negations = pd.DataFrame({'pred':[predict(i) for i in negations_test['text']],'pred_negations':[predict_negation(i) for i in negations_test['text']], 'true':negations_test['polarity'],'text':negations_test['text']})\n",
    "result_original = pd.DataFrame({'pred':[predict(i) for i in df_test['text']],'pred_negations':[predict_negation(i) for i in df_test['text']], 'true':df_test['polarity'],'text':df_test['text']})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "     positiv       0.62      0.53      0.57        40\n",
      "     negativ       0.47      0.42      0.45        40\n",
      "\n",
      "   micro avg       0.54      0.47      0.51        80\n",
      "   macro avg       0.54      0.47      0.51        80\n",
      "weighted avg       0.54      0.47      0.51        80\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "     positiv       0.57      0.85      0.68        40\n",
      "     negativ       0.70      0.35      0.47        40\n",
      "\n",
      "    accuracy                           0.60        80\n",
      "   macro avg       0.63      0.60      0.57        80\n",
      "weighted avg       0.63      0.60      0.57        80\n",
      "\n"
     ]
    }
   ],
   "source": [
    "result_negations['true'] = result_negations['true'].replace([0,2,1],['positiv','negativ','neutral'])\n",
    "\n",
    "print(classification_report(result_negations['true'],result_negations['pred'],labels=['positiv','negativ']))\n",
    "print(classification_report(result_negations['true'],result_negations['pred_negations'],labels=['positiv','negativ']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we see that the effect of training with more negation sentences is ambigious. Even though the averaged performance is better, it seems to have skewed the prediction to positive as the recall is higher for the positive sentiment in the model trained on negation sentences. Therefore, the regulation might have worked too well in accepting negations in sentences with positive sentiment. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We do also test the performance on the original test set to evaluate if the negations has improved the general performance of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "     negativ       0.64      0.87      0.74       271\n",
      "     neutral       0.48      0.19      0.27       120\n",
      "     positiv       0.53      0.40      0.46       121\n",
      "\n",
      "    accuracy                           0.60       512\n",
      "   macro avg       0.55      0.49      0.49       512\n",
      "weighted avg       0.58      0.60      0.56       512\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "     negativ       0.64      0.85      0.73       271\n",
      "     neutral       0.50      0.14      0.22       120\n",
      "     positiv       0.48      0.47      0.47       121\n",
      "\n",
      "    accuracy                           0.59       512\n",
      "   macro avg       0.54      0.49      0.48       512\n",
      "weighted avg       0.57      0.59      0.55       512\n",
      "\n"
     ]
    }
   ],
   "source": [
    "result_original['true'] = result_original['true'].replace([0,2,1],['positiv','negativ','neutral'])\n",
    "\n",
    "print(classification_report(result_original['true'],result_original['pred']))\n",
    "print(classification_report(result_original['true'],result_original['pred_negations']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we see the tendency of a higher recall on positive sentiment. Apart from that, the negations haven't changed much in performance of the model. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}