examples/tutorials/example_zero_shot_sentiment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Zero Shot Sentiment Analysis using LASER\n",
    "\n",
    "This is a notebook to show an example of how NLP methods can utilize data in another language to perform a task in Danish with no need for Danish data. Imagine a task where there is no annotated data in Danish but there exists open source dataset in another language for a similar task. In such a case it is sometimes possible to train a model on data from the so called 'source' language an imply it straight away on the 'target' language. This is called  **zero-shot transfer** when the classifier has not seen any data from the target language.\n",
    "\n",
    "The overall idea is to create a multilingual embedding space to represent words or sentences from several languages in a similar matter. This means that we can take a sentence in one language and map it into a vector representation which basically is an array of numbers. If we then have a sentence in another language with the same meaning, we can then map this sentence as well into the vector representation. This would ideally give us to arrays of numbers which are very similar. We can also think of it in a geometrically sense; these vectors lie close to each other in our multilingual embeddings space.\n",
    "\n",
    "The idea is now to map the annotated data in the source language into such representations. We can then use these representations as features and train a (simple) classification model. When we want to apply the model on the target language, we simply take the input sentences and map it into the representation and then apply the trained classifier.\n",
    "\n",
    "In this example we will be working with  **LASER sentence embeddings** from Facebook Research. Have a look at their [github](https://github.com/facebookresearch/LASER) or read the paper for further understanding: Holger Schwenk and Matthijs Douze, [Learning Joint Multilingual Sentence Representations with Neural Machine Translation](https://aclweb.org/anthology/papers/W/W17/W17-2619/), ACL workshop on Representation Learning for NLP, 2017.\n",
    "The Laser embeddings are trained using machine translation on 93 languages using a shared encoder, and the data is different sets of parallel corpuses that translate into English and Spanish.\n",
    "\n",
    "The task we will look at in this example is **Sentiment Analyses** performed on data from movie reviews. We will be using the IMDB dataset for training, and then we will be testing on a Norwegian dataset.\n",
    "\n",
    "\n",
    "*NOTE:\n",
    "The code in this notebook is not integrated in the DaNLP, but we are working on a Danish dataset for sentiment and with it a model for sentiment analyse, that is properly benchmarked*\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The steps \n",
    "**pre-steps**\n",
    "1. Download the data and extract it to the right format\n",
    "2. Install the libraries needed\n",
    "\n",
    "**steps inside the Jupyter Notebook**\n",
    "3. Prepare and clean the data, and embed it using LASER\n",
    "5. Train a classifier\n",
    "6. Try it on Danish text\n",
    "6. Evaluate on target language data - in this case a Norwegian corpus\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get the data\n",
    "\n",
    "**The IMDB dataset**\n",
    "Download the data at http://ai.stanford.edu/~amaas/data/sentiment/\n",
    "and cite the paper:\n",
    "Maas, Andrew L.  and  Daly, Raymond E.  and  Pham, Peter T.  and  Huang, Dan  and  Ng, Andrew Y.  and  Potts, Christopher, [Learning Word Vectors for Sentiment Analysis](http://www.aclweb.org/anthology/P11-1015), ACL 2011\n",
    "\n",
    "The data consists of 50K reviews from IMDB, and the labels originate from the ratings turned into a binary classification task. The data is split equally into a test set and in a trainset. The data is balanced between the two classes.\n",
    "\n",
    "Once the data is downloaded, we will combine the training and testing part into one txt file each. It can be done in the following manner (from this [blogpost](https://towardsdatascience.com/sentiment-analysis-with-python-part-1-5ce197074184)), open a terminal and navigate to the download file, named: aclImdb_v1.tar.gz\n",
    "\n",
    "Unzip:\n",
    "`gunzip -c aclImdb_v1.tar.gz | tar xopf -`\n",
    "\n",
    "Navigate and make new folder: `cd aclImdb && mkdir movie_data`\n",
    "\n",
    "Concetenate into a txt file: `for split in train test; do for sentiment in pos neg; do for file in $split/$sentiment/*; do cat $file >> movie_data/full_${split}.txt; echo >> movie_data/full_${split}.txt; done; done; done;` \n",
    "\n",
    "Now there will be a file named full_train.txt and a file named full_test.txt in the folder aclImdb/movie_data.\n",
    "\n",
    "\n",
    "**NoReC: The Norwegian Review Corpus**\n",
    "The data contains reviews from different domains including movies originating from Norwegian news sources. The information of each reviews origins is stored as metadata along with ratings. The ratings are made comparable across domains and are in the range between 1 and 6. The dataset is split into train, validation and test. Read more about the data in the paper [NoReC: The Norwegian Review Corpus](http://www.lrec-conf.org/proceedings/lrec2018/pdf/851.pdf) Erik Velldal, Lilja Øvrelid, Eivind Alexander Bergem, Cathrine Stadsnes, Samia Touileb, Fredrik Jørgensen, 2018\n",
    "\n",
    "Clone the github to get the data,\n",
    "\n",
    "`git clone https://github.com/ltgoslo/norec`\n",
    "\n",
    "`cd norec`\n",
    "\n",
    "`./download.sh`\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup the installation\n",
    "You need the following python packages to run the code in this notebook. It is recommended to install it through a virtual environment, for example use pip [read more here](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/).\n",
    "\n",
    "The NoRec github include a package for extrating the text. Go to the folder 'src' and follow the instruction to install the package norec, [see here](https://github.com/ltgoslo/norec/tree/master/src).\n",
    "\n",
    "The following packages can be installed thorugh pip:\n",
    "\n",
    "- [laserembeddings](https://pypi.org/project/laserembeddings/) - This package is a wrapper of the Laser embeddings integrated to work directly in a python script, but feel free to use the original souce code from Laser.\n",
    "\n",
    "- [scikit-learn](https://pypi.org/project/scikit-learn/) - This is used to fit a classification model\n",
    "- NumPy\n",
    "- Pandas\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import libaries\n",
    "import re\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pickle\n",
    "import norec\n",
    "\n",
    "from laserembeddings import Laser\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import accuracy_score, log_loss\n",
    "from sklearn.model_selection import train_test_split\n",
    "from itertools import product\n",
    "from sklearn.metrics import confusion_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1:  Prepare the data \n",
    "\n",
    "First prepare the training data from the IMDB dataset such it is ready for the laser embeddings. We will only work with the training part. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"I enjoyed The Night Listener very much. It's one of the better movies of the summer.<br /><br />Robin Williams gives one of his best performances. In fact, the entire cast was very good. All played just the right notes for their characters - not too much and not too little. Sandra Oh adds a wonderful comic touch. Toni Collette is great as the Mom, and never goes over the top. Everyone is very believable.<br /><br />It's a short movie, just under an hour and a half. I noticed the general release version is nine minutes shorter than the Sundance version. I wonder if some of the more disturbing images were cut from the movie.<br /><br />The director told a story and did it in straightforward fashion, which is a refreshing change from many directors these days who seem to think their job is to impress the audience rather than tell a story and tell it well.<br /><br />Do not be sucker punched by the previews and ads. It is not a Hitchcockian thriller. See The Night Listener because you want to see a good story told well. If you go expecting Hitchcock you will be disappointed.<br /><br />My only complaint with the movie was the ending. The director could have left a little more to the audience's imagination, but this is a minor quibble.\""
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load the review text into a list\n",
    "path = 'aclImdb/movie_data/full_train.txt' # here set the path to the full_train.txt file \n",
    "\n",
    "reviews_train = []\n",
    "for line in open(path, 'r'):\n",
    "    reviews_train.append(line.strip())\n",
    "\n",
    "# lets have a look at one abitra review\n",
    "reviews_train[13]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I enjoyed The Night Listener very much. Its one of the better movies of the summer. Robin Williams gives one of his best performances. In fact, the entire cast was very good. All played just the right notes for their characters   not too much and not too little. Sandra Oh adds a wonderful comic touch. Toni Collette is great as the Mom, and never goes over the top. Everyone is very believable. Its a short movie, just under an hour and a half. I noticed the general release version is nine minutes shorter than the Sundance version. I wonder if some of the more disturbing images were cut from the movie. The director told a story and did it in straightforward fashion, which is a refreshing change from many directors these days who seem to think their job is to impress the audience rather than tell a story and tell it well. Do not be sucker punched by the previews and ads. It is not a Hitchcockian thriller. See The Night Listener because you want to see a good story told well. If you go expecting Hitchcock you will be disappointed. My only complaint with the movie was the ending. The director could have left a little more to the audiences imagination, but this is a minor quibble.'"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The reviews need to be clean for different xml tags\n",
    "REMOVE = re.compile(\"(<br\\s*/><br\\s*/>)|(\\-)|(\\/)\")\n",
    "\n",
    "def preprocess_reviews(reviews):\n",
    "    reviews = [REMOVE .sub(\" \", line) for line in reviews]\n",
    "    reviews = [line.replace('\\'','') for line in reviews]\n",
    "    return reviews\n",
    "\n",
    "reviews_train_clean = preprocess_reviews(reviews_train)\n",
    "\n",
    "# and lets have a look again\n",
    "reviews_train_clean[13] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Embed with Laser\n",
    "Now the reviews are ready to be embedded using Laser. In this first example we will take each review and embed it into one vector.  The embeddings will have a dimension of 1024 for each input, and the Laser embedding needs the input text, and the languages used for tokenization. Let us start with an example of using Laser."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 1024)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# run an example to see it is working\n",
    "laser = Laser()\n",
    "examples = ['Det kunne være fedt med en sentiment klassifier på dansk!', 'Lad os prøve med en zero shot tilgang.' ]\n",
    "embeddings = laser.embed_sentences(examples, lang='da')\n",
    "embeddings.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now embed all the reviews in the training data \n",
    "# note this might take a really long time \n",
    "embedings = laser.embed_sentences(reviews_train_clean, lang='en')\n",
    "\n",
    "# check you got the expected output\n",
    "embedings.shape # this shoud be (25.000, 1024)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the embeddings, then if you return to the notebook later there will be no need to run STEP 1 again\n",
    "path_imdb_embeddings = 'aclImdb/movie_data/imdb_clean_train_laser' # choose the path to store the embedded imdb reviews\n",
    "np.save(embedings, path_imdb_embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Train a classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the embeddings of the reviews, which is the feature vector to train the classifier \n",
    "path_imdb_embeddings = 'aclImdb/movie_data/imdb_clean_train_laser' # choose the path where the embedded imdb reviews are stored\n",
    "\n",
    "# load the training features\n",
    "features = np.load(path_imdb_embeddings + '.npy')\n",
    "\n",
    "# the tager vector - from the way the reviews were concatenated into one file, we have that the\n",
    "target = [1 if i < 12500 else 0 for i in range(25000)]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Randomize the order, and split into train and validation set\n",
    "X_train, X_val, y_train, y_val = train_test_split(features, target, test_size = 0.10, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on val: 0.796 with c value 0.1, penalty l2 and solver lbfgs \n",
      "Accuracy on val: 0.8288 with c value 1, penalty l2 and solver lbfgs \n",
      "Accuracy on val: 0.8348 with c value 10, penalty l2 and solver lbfgs \n",
      "Accuracy on val: 0.8364 with c value 100, penalty l2 and solver lbfgs \n"
     ]
    }
   ],
   "source": [
    "# train a model using logistic regression\n",
    "\n",
    "# logistic regression - Try different values\n",
    "solvers = ['lbfgs'] #['newton-cg', 'lbfgs', 'liblinear', 'sag', 'saga']\n",
    "regulation = ['l2']\n",
    "c_values = [0.1,1,10,100]\n",
    "for c,l,s in product(c_values, regulation,solvers): # with product we can iterate through all possible combinations \n",
    "    lr = LogisticRegression(C=c, solver=s, penalty=l, random_state=42, max_iter=10000000)\n",
    "    lr.fit(X_train, y_train)\n",
    "    print (\"Accuracy on val: %s with c value %s, penalty %s and solver %s \" \n",
    "           % ( accuracy_score(y_val, lr.predict(X_val)),c,l,s))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1074,  224],\n",
       "       [ 185, 1017]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# lets examine if the classifier is better to predict one class or not\n",
    "y_pred = lr.predict(X_val)\n",
    "confusion_matrix(y_val, y_pred, labels=[1,0]) # 1 is positive and 0 is negative\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the choosen classifier model to disk\n",
    "path_model = 'aclImdb/movie_data/Lr_model.sav'\n",
    "pickle.dump(lr, open(path_model, 'wb'))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3:  Apply on Danish Exampels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# if nesesary load the model from disk which was build above\n",
    "path_model = 'aclImdb/movie_data/Lr_model.sav'\n",
    "lr = pickle.load(open(path_model, 'rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# creat a function to judge the sentiment on danish sentences\n",
    "def get_sentiment(sentence, classifier, token_lang='da'):\n",
    "    \n",
    "    # embed the sentence\n",
    "    laser = Laser()\n",
    "    input_features = laser.embed_sentences([sentence], lang=token_lang)\n",
    "    \n",
    "    # apply the classifier\n",
    "    pred=classifier.predict(input_features)\n",
    "    \n",
    "    class_names = {'0': 'negative', '1': 'positve'}\n",
    "    \n",
    "    return class_names[str(int(pred))]\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'negative'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's try with some examples\n",
    "get_sentiment('Det var ikke godt', lr)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'positve'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's try with some examples\n",
    "get_sentiment('Det var godt, ikke?', lr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'negative'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's try with some movie examples\n",
    "get_sentiment('Filmen svarer til at kigge ind i en hvid væg i to timer', lr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'positve'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's try with some movie examples with mix of languages\n",
    "get_sentiment('Musikalfilmen Les Mesirable er min ynglings fordi den er just fabulous', lr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'positve'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's try to give it a harder example, and see it fail\n",
    "get_sentiment('Jeg så filmen sammen med mine dejlige veninder, men det var også det eneste gode at sige om den film', lr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comments\n",
    "Note that the transfer performance is not only depended on the language we transfer from and too, but also on the domain used to train the classifier. A shift in domain - in this cause to other than movie reviews from IMDB - would also affect the classifier's performance. Likewise, using the classifier on data which is more clear in polarity would give higher result in using it on data that a more refined in its sentiment. \n",
    "\n",
    "The next step is to try to evaluate den model on a Norweign corpus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4:  Evaluate on Norwegian Data\n",
    "\n",
    "Presteps: follow the instruction in the top of the notebook to clone the Norec repository, download the data, and install the norec package. \n",
    "\n",
    "Now we will first prepare the data. We will be working with the subset defined as 'train' to test our model, since this is the larges subset. To ensemble the IMDB dataset we will make the task binary by dropping reviews with ratings in the middle (3 and 4), and combine the reviews with ratings 5 and 6 to positive, and the ones with 1 and 2 to negative. Further more, we will sample only the 'movie' reviews to remain in a similar domain as the training data from IMDB. Lastly we will balance the reviews to include equal number of positive and negative. \n",
    "\n",
    "Then we embed these reviews with LASER.\n",
    "\n",
    "And then we test our train classifier. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to prepare the data using the NOREC package\n",
    "\n",
    "\n",
    "def prepare_data(subset, path):\n",
    "    # subset is either: 'train', 'dev' or 'test'\n",
    "    # path to the html file \"norec/data/html.tar.gz\"\n",
    "    # load the data\n",
    "    subset_data = norec.load(path, subset=subset)\n",
    "    \n",
    "    # Get the reviews, ratings and the sub katagori\n",
    "    subset_list = [(norec.html_to_text(html), metadata['rating'], metadata['source-category'])\n",
    "                 for html, metadata in subset_data]\n",
    "\n",
    "    df  = pd.DataFrame(subset_list, columns=['reviews', 'score', 'genre'])\n",
    "    \n",
    "    # Keep only the movie revies\n",
    "    df=df[df['genre']=='film']\n",
    "\n",
    "    # drop reviews with score 4 or 3\n",
    "    df = df[df.score != 3]\n",
    "    df = df[df.score != 4]\n",
    "\n",
    "    df.loc[df['score'] < 3, 'score'] = 0\n",
    "    df.loc[df['score'] > 3, 'score'] = 1\n",
    "\n",
    "    # clan reviews\n",
    "    df['reviews']=df['reviews'].apply(lambda x: x.replace(\"\\n\", ' '))\n",
    "\n",
    "    # make a balance set\n",
    "    count_labels = pd.Series(df['score']).value_counts()\n",
    "\n",
    "    if count_labels[1] > count_labels[0]:\n",
    "        large_class = 'score == 1' \n",
    "        drop_fraction = 1-count_labels[0]/count_labels[1]\n",
    "    else: \n",
    "        drop_fraction = 1-count_labels[1]/count_labels[0]\n",
    "        large_class = 'score == 0' \n",
    "    df=df.drop(df.query(large_class).sample(frac=drop_fraction).index)\n",
    "    \n",
    "    return df['reviews'], df['score']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Thumbsucker  Han er flink men har ikke venner. Han er med i et team som heter delta team. Moren (Tilda Swinton) hans er sykepleier hun får jobben på et nytt sykehus der hun blir forelsket i en pasient som er  Kjendis. Faren(Vincent d’Onofrio) jobbet i en sportsbutikk han hadde  alltid tapt i løp mot tannlegen til (Keanu Reeves) Justin prøver å slut-te å suge på tommeltotten hans. Han klarer det til slutt.  anmeldelse av patrick rambjør\n",
      "(2224,)\n"
     ]
    }
   ],
   "source": [
    "# prepare the data\n",
    "path_norec_data = \"norec/data/html.tar.gz\" # rember to set the right path\n",
    "reviews_norwe, y_norwe = prepare_data('train',path_norec_data)\n",
    "\n",
    "# print an example of a review\n",
    "print(reviews_norwe.iloc[1]) # note have the text is more descriping the plot in the movie instead of giving opinions of it\n",
    "\n",
    "# print the numbers of revies\n",
    "print(reviews_norwe.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# embed with Laser\n",
    "# this can take along time\n",
    "laser = Laser()\n",
    "X_norwe= laser.embed_sentences(reviews_norwe, lang='no')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on Norweigian data: 0.7630395683453237 \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[926, 186],\n",
       "       [341, 771]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now use the model 'lr' to predict and calculate the accuracy\n",
    "y_pred = lr.predict(X_norwe)\n",
    "print (\"Accuracy on Norweigian data: %s \" \n",
    "       % ( accuracy_score(y_norwe, y_pred)))\n",
    "\n",
    "# Let's seen the confussion matric \n",
    "confusion_matrix(y_norwe, y_pred, labels=[1,0])   # 1 is positive and 0 is negative\n",
    "# Note that it looks like the classifier has a harder time predicting the negative class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}