examples/tutorials/example_text_classification.ipynb

{
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5-final"
  },
  "orig_nbformat": 2,
  "kernelspec": {
   "name": "python38564bitdanlpcondab70fe6e4720d408985175c9cab14fc3e",
   "display_name": "Python 3.8.5 64-bit ('danlp': conda)"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2,
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text classification of speeches from the Danish Parliament\n",
    "\n",
    "In this notebook, we show how to perform text classification. \n",
    "As a concrete example, we train a text classifier to predict the party in which belongs a speech from the Danish Parliament. \n",
    "\n",
    "Steps in this tutorial:\n",
    "\n",
    "1. Download the data\n",
    "2. Extract the texts from the XML files\n",
    "3. Preprocess the data (cleaning)\n",
    "4. Prepare the data for training and evaluation\n",
    "5. Train a model with fastText\n",
    "6. Test and evaluate the performance\n",
    "\n",
    "You can restart the tutorial at each step if you have previously saved the models/data. \n",
    "To reinitialize the notebook with required libraries and variables, run the following cell. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell for restarting from any step\n",
    "\n",
    "import os\n",
    "import glob\n",
    "from zipfile import ZipFile\n",
    "import xmltodict\n",
    "import timestring\n",
    "import json\n",
    "import fasttext\n",
    "\n",
    "# path to the (default) data folder \n",
    "# where we save/load data and models for this notebook \n",
    "DATA_DIR = \"FT-data-DSpace\"\n",
    "\n",
    "# paths to the training and test files\n",
    "train_path = os.path.join(DATA_DIR, \"train.txt\")\n",
    "test_path = os.path.join(DATA_DIR, \"test.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1. Download the Data\n",
    "\n",
    "The data we use comes from the `The Danish Parliament Corpus 2009 - 2017, v1` (Hansen, Dorte Haltrup, 2018, CLARIN-DK-UCPH Centre Repository).\n",
    "\n",
    "The corpus contains transcripts of parliamentary speeches. It consists of 10 XML files (one for each year). XML tags include meetings, item title and number, speeches, name and party of speakers, date, time, etc.\n",
    "\n",
    "\n",
    "1. Download the data at http://hdl.handle.net/20.500.12115/8\n",
    "2. Unzip the folder : `unzip FT-data-DSpace.zip`\n",
    "3. Enter the folder and unzip the files : `cd FT-data-DSpace && unzip '*.zip'`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2. Extract the texts\n",
    "\n",
    "For building and testing a model, we need labelled (classified) texts. \n",
    "Texts are transcripts of speeches; labels/classes are the parties they belong to. \n",
    "We start by extracting these data from the XML files. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each parliament year, we want to extract all the speeches that are attached to a party (some speeches have no party attached, we don't keep them in our data) and store them as a list of triplets (year, party, speech). \n",
    "In the xml file, speeches are stored under the `<EdixiData><Møder><Møde><Dagsordenpunkt><Tale>` tags. The recorded year, name of the party and transcript of the speech are respecctively stored with the `<Starttid>`, `<Parti>` and `<Tekst>` tags. \n",
    "The structure of an XML file is as follows: \n",
    "\n",
    "~~~ xml\n",
    "    <EdixiData>\n",
    "        <Møder>\n",
    "            <Samling>...</Samling>\n",
    "            <Møde>\n",
    "                <MeetingID>...</MeetingID>\n",
    "                <Location>...</Location>\n",
    "                <DateOfSitting>...</DateOfSitting>\n",
    "                <Mødenummer>...</Mødenummer>\n",
    "                <Dagsordenpunkt>\n",
    "                    <Punktnummer>...</Punktnummer>\n",
    "                    <Mødetitle>...</Mødetitle>\n",
    "                    <Sagstype>...</Sagstype>\n",
    "                    <Tale>\n",
    "                        <Starttid>...</Starttid>\n",
    "                        <Sluttid>...</Sluttid>\n",
    "                        <Navn>...</Navn>\n",
    "                        <Rolle>...</Rolle>\n",
    "                        <Tekst>...</Tekst>\n",
    "                    </Tale>\n",
    "                    <Tale>\n",
    "                    ...\n",
    "                    </Tale>\n",
    "                </Dagsordenpunkt>\n",
    "            </Møde>\n",
    "            ...\n",
    "        </Møder>\n",
    "    </EdixiData>\n",
    "~~~\n",
    "\n",
    "\n",
    "We first define a function for extracting this data from one xml file. \n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def speeches_from_xml(xml_file):\n",
    "\n",
    "    from collections import OrderedDict\n",
    "\n",
    "    speeches = []\n",
    "\n",
    "    # converting xml structure to dict\n",
    "    xml_data = xmltodict.parse(xml_file.read())\n",
    "    xml_data = xml_data['EdixiData']['Møder']\n",
    "\n",
    "    for moder in xml_data:\n",
    "        if not isinstance(moder, OrderedDict):\n",
    "            continue\n",
    "        if not 'Møde' in moder.keys():\n",
    "            continue\n",
    "        for meeting in moder['Møde']:\n",
    "            if not isinstance(meeting, OrderedDict):\n",
    "                continue\n",
    "            for dagsordenpunkt in meeting['Dagsordenpunkt']:\n",
    "                if not isinstance(dagsordenpunkt, OrderedDict):\n",
    "                    continue\n",
    "                if not 'Tale' in dagsordenpunkt:\n",
    "                    continue\n",
    "                for tale in dagsordenpunkt['Tale']:\n",
    "                    if not isinstance(tale, OrderedDict):\n",
    "                        continue\n",
    "                    if not 'Parti' in tale or not isinstance(tale['Parti'], str):\n",
    "                        continue\n",
    "                    if not 'Starttid' in tale or not isinstance(tale['Starttid'], str):\n",
    "                        continue\n",
    "                    if not 'Tekst' in tale or not isinstance(tale['Tekst'], str):\n",
    "                        continue\n",
    "                    # we only save the year, not the exact date of the speech\n",
    "                    year = str(timestring.Date(tale['Starttid']).year)\n",
    "                    party = tale['Parti']\n",
    "                    text = tale['Tekst']\n",
    "                    if len(text)<1:\n",
    "                        continue\n",
    "                    speeches.append({'year': year, 'party': party, 'text': text})\n",
    "\n",
    "    return speeches"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We run it on all the XML files, extracting the speeches from 2009 to 2017."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "speeches = []\n",
    "for xml_path in glob.glob(os.path.join(DATA_DIR, \"EdixiXMLExport_*.zip\")):\n",
    "    filename = os.path.splitext(os.path.basename(xml_path))[0]\n",
    "    print(\"Extract texts from \", filename)\n",
    "    with ZipFile(xml_path) as xml_zip:\n",
    "        with xml_zip.open(filename+'.xml') as xml_file:\n",
    "            speeches += speeches_from_xml(xml_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(\"Example\\n-------\")\n",
    "print(speeches[0])\n",
    "print()\n",
    "\n",
    "# listing the years\n",
    "years = sorted(list(set([s['year'] for s in speeches])))\n",
    "for year in years:\n",
    "    print(len([_ for s in speeches if s['year']==year]), \" speeches in \", year)\n",
    "print()\n",
    "\n",
    "# listing the parties \n",
    "parties = sorted(list(set([s['party'] for s in speeches])))\n",
    "for party in parties:\n",
    "    print(len([_ for s in speeches if s['party']==party]), \" speeches from \", party)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We save the data so we can restart the notebook from the preprocessing step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join(DATA_DIR, \"speeches.json\"), 'w') as f:\n",
    "    f.write(json.dumps(speeches, indent=4))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3. Preprocess (clean) the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For preprocessing the texts, we use the Danish SpaCy model.\n",
    "Using this model, we can tokenize and tag the sentences with part-of-speech. \n",
    "\n",
    "First, we load the spacy model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from danlp.models import load_spacy_model\n",
    "nlp = load_spacy_model()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We load the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "speeches = []\n",
    "with open(os.path.join(DATA_DIR, \"speeches.json\")) as f:\n",
    "    speeches = json.loads(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And preprocess (clean) the texts by: \n",
    "- removing punctuation and symbols\n",
    "- removing stop words and numbers\n",
    "- lowercasing the tokens\n",
    "\n",
    "(This process might take several minutes)\n",
    "\n",
    "The purpose of this step is to reduce the vocabulary in order to speed up the training process. It is possible to skip some of the cleaning steps in order to improve the quality of the prediction (e.g., lowercasing might reduce the benefits of using word embeddings). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "outputPrepend"
    ]
   },
   "outputs": [],
   "source": [
    "from stop_words import get_stop_words\n",
    "da_stopwords = get_stop_words('da')\n",
    "\n",
    "import lemmy.pipe\n",
    "lemmatizer = lemmy.load('da')\n",
    "\n",
    "for speech in speeches:\n",
    "    text = speech['text']\n",
    "    doc = nlp(text)\n",
    "    pruned = []\n",
    "    lemmas = []\n",
    "    for tok in doc:\n",
    "        if tok.tag_ in [\"PUNCT\", \"SYM\"]:\n",
    "            continue\n",
    "        if tok.is_stop or tok.is_digit:\n",
    "            continue\n",
    "        pruned.append(tok.lower_)\n",
    "        lemmas.append(lemmatizer.lemmatize(tok.tag_, tok.lower_)[0])\n",
    "\n",
    "    speech['preprocessed'] = \" \".join(pruned)\n",
    "    speech['lemmas'] = \" \".join(lemmas)\n",
    "\n",
    "speeches[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    " with open(os.path.join(DATA_DIR, \"speeches_pp.json\"), 'w') as f:\n",
    "    f.write(json.dumps(speeches, indent=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4. Prepare the data for training and testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "We re-load the preprocessed data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "speeches = []\n",
    "with open(os.path.join(DATA_DIR, \"speeches_pp.json\")) as f:\n",
    "    speeches = json.loads(f.read())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We split into train and test data.\n",
    "We will build a model from speeches from 2009 to 2014 and evaluate its performance on the 2015 speeches. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "\n",
    "train_data = [(sp['party'],sp['preprocessed']) for sp in speeches if not sp['year'] == '2015']\n",
    "test_data = [(sp['party'],sp['preprocessed']) for sp in speeches if sp['year'] == '2015']\n",
    "\n",
    "print(\"Training data : \", len(train_data), \"speeches\")\n",
    "print(\"Test data : \", len(test_data), \"speeches\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We save the training and test data in a format accepted by fastText : \n",
    "```\n",
    "__label__class1 text1\n",
    "__label__class2 text2\n",
    "...\n",
    "__label__classN textN\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "\n",
    "with open(train_path, 'w') as f:\n",
    "    for (p,t) in train_data:\n",
    "        f.write(\"__label__\"+p+\" \"+t+\"\\n\")\n",
    "\n",
    "with open(test_path, 'w') as f:\n",
    "    for (p,t) in test_data:\n",
    "        f.write(\"__label__\"+p+\" \"+t+\"\\n\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5. Learn a model with fastText"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We load the common crawl word embeddings (\"`cc.da.wv`\") from fastText using Gensim. \n",
    "If you prefer to use other embeddings from our library, you can have a look at our [page](https://github.com/alexandrainst/danlp/blob/master/docs/models/embeddings.md).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from danlp.models.embeddings import load_wv_with_gensim\n",
    "wv = load_wv_with_gensim(\"cc.da.wv\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We save the embeddings in a format that is accepted by fastText (i.e. .vec)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wv.save_word2vec_format(os.path.join(DATA_DIR, \"cc-wv.vec\"), binary=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We train the model with fastText. We can fine-tune the hyperparameters, e.g.:\n",
    "- the number of epochs (recommended : from 1 to 50)\n",
    "- the learning rate (recommended : from 0.1 to 1.0)\n",
    "- the N-grams length (recommended : from 1 to 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import time \n",
    "\n",
    "start_time = time.time()\n",
    "model = fasttext.train_supervised(input=train_path, epoch=20, lr=0.2, dim=100, wordNgrams=2, pretrainedVectors=os.path.join(DATA_DIR, \"cc-wv.vec\"))\n",
    "print(\"time :\", time.time()-start_time)\n",
    "print(\"score : \", model.test(test_path))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save_model(os.path.join(DATA_DIR, \"model.bin\"))"
   ]
  },
  {
   "source": [
    "## Step 6. Test and Evaluate the performance"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "Load the model (and test on the test data)."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model = fasttext.load_model(os.path.join(DATA_DIR, \"model.bin\"))\n",
    "model.test(test_path)"
   ]
  },
  {
   "source": [
    "To make a prediction with the model, you can use the following code (you can replace the text with any (preprocessed) text) : "
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "\n",
    "text = \"håber enighed gennemføre saglig seriøs høring hele lovforslaget dets aspekter tror element frank aaen nævner vedkommende volde umiddelbart største problemer\"\n",
    "predicted_party = model.predict(text)\n",
    "predicted_party = predicted_party[0][0].split(\"__\")[-1]\n",
    "print(predicted_party)"
   ]
  },
  {
   "source": [
    "We calculate the accuracy per label to see how the model performs for each party and the micro average accuracy."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "golds = []\n",
    "preds = []\n",
    "with open(test_path) as f:\n",
    "    for line in f:\n",
    "        line = line.strip()\n",
    "        line = line.split(\" \", maxsplit=1)\n",
    "        text = line[1]\n",
    "        gold_party = line[0].split(\"__\")[-1]\n",
    "        pred_party = model.predict(text)[0][0].split(\"__\")[-1]\n",
    "        golds.append(gold_party)\n",
    "        preds.append(pred_party)\n",
    "\n",
    "golds = np.array(golds)\n",
    "preds = np.array(preds)\n",
    "labels = np.unique(golds)\n",
    "cf = confusion_matrix(golds, preds, labels=labels)\n",
    "\n",
    "print(\"Accuracy per party\")\n",
    "for i, label in enumerate(labels):\n",
    "    tp = cf[i][i]\n",
    "    tt = np.sum(golds == label)\n",
    "    print(label, \"\\t{:.1%}\\t({}/{})\".format(tp/tt, tp, tt))\n",
    "\n",
    "print(\"\\nGlobal accuracy\")\n",
    "tp = np.sum(preds==golds)\n",
    "tt = len(golds)\n",
    "print(\"{:.1%}\\t({}/{})\".format(tp/tt, tp, tt))"
   ]
  }
 ]
}