{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine learning with scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Georgios Gousios <g.gousios@tudelft.nl>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be predicting whether pull requests are merged, using feature engineering and classic macine learning approaches. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we make sure the data is in place"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "\n",
    "if not os.path.exists(\"../datasets/scala.csv\"):\n",
    "  url = \"https://raw.githubusercontent.com/gousiosg/pullreqs/783aef46a3f1cde5663dd985325f5ee069bf841a/data/scala%40scala.csv\"\n",
    "  data = pd.read_csv(url)\n",
    "  data.to_csv(\"../datasets/scala.csv\")\n",
    "else:\n",
    "  data = pd.read_csv(\"../datasets/scala.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas reads a data frame directly from a URL, which saves us lots of trouble. Let's see what the data looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['Unnamed: 0', 'pull_req_id', 'project_name', 'lang', 'github_id',\n",
      "       'created_at', 'merged_at', 'closed_at', 'lifetime_minutes',\n",
      "       'mergetime_minutes', 'merged_using', 'conflict', 'forward_links',\n",
      "       'team_size', 'num_commits', 'num_commit_comments', 'num_issue_comments',\n",
      "       'num_comments', 'num_participants', 'files_added', 'files_deleted',\n",
      "       'files_modified', 'files_changed', 'src_files', 'doc_files',\n",
      "       'other_files', 'perc_external_contribs', 'sloc', 'src_churn',\n",
      "       'test_churn', 'commits_on_files_touched', 'test_lines_per_kloc',\n",
      "       'test_cases_per_kloc', 'asserts_per_kloc', 'watchers', 'requester',\n",
      "       'prev_pullreqs', 'requester_succ_rate', 'followers', 'intra_branch',\n",
      "       'main_team_member'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "print(data.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we are doing supervised learning, we need a label to supervise our training upon. We construct it below, as a binary feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['merged'] = data.merged_at.map(lambda x: 0 if pd.isna(x) else 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see our data distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    2753\n",
       "0     470\n",
       "Name: merged, dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.merged.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our data is skewed towards the True (1) label. Actually, 85% of our labels are True. This means that if we build a classifier that always reports True as the output label, it will be correct 85% of the time. We have to beat that! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To simplify our training, we will only be using numerical features. Binary (or multiclass) features require a transformation called one-hot encoding, which we leave as an exercise to the reader."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "columns = [\"team_size\", \"num_commits\", \"files_changed\", \"perc_external_contribs\",\n",
    "  \"num_comments\", \"commits_on_files_touched\",  \"test_lines_per_kloc\",\n",
    "  \"prev_pullreqs\", \"requester_succ_rate\", \"num_participants\"]\n",
    "\n",
    "X = data[columns].to_numpy()\n",
    "y = data.merged.to_numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For evaluating our predictors, we need to split our data into training and test sets. Otherwise, we will be training and evaluating on the same data, which our classifier might have learned \"by heart\" (i.e. overfitted). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first train a [binary logistic regression](https://en.wikipedia.org/wiki/Logistic_regression) classifier; it attempts to find the values for the coefficients of a linear function of the feature values to the output variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "blr = LogisticRegression(random_state=0, solver=\"lbfgs\", max_iter=1000).\\\n",
    "  fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the model is trained, we can get the coefficients for all the features. If we order them, we can a sense of how imporant each feature is on the output prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "requester_succ_rate: 0.7707932567056638\n",
      "commits_on_files_touched: 0.002985201513517729\n",
      "num_comments: 0.00287620153754614\n",
      "prev_pullreqs: 0.000972958806073166\n",
      "files_changed: -0.00286068837336261\n",
      "team_size: -0.008821429118866771\n",
      "num_commits: -0.010450606903349468\n",
      "perc_external_contribs: -0.011170488139997025\n",
      "test_lines_per_kloc: -0.01956302907174823\n",
      "num_participants: -0.09039807198121654\n"
     ]
    }
   ],
   "source": [
    "for feature, coef in sorted(zip(columns, blr.coef_[0]), key=lambda x: x[1], reverse=True):\n",
    "    print(\"{}: {}\".format(feature, coef))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try to predict with our trained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_predicted = blr.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_predicted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hmmmm.... Lots of 1s. This means that the model probably overfitted. But how can we evaluate the model's performance? Let's use precision and recall, the standard metrics for classification performance:\n",
    "\n",
    "* Precision tells us of the predictions the model reported as True, how many are actually True.\n",
    "* Recall tells us the % of the True labels the model found."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "precision:1.0\n",
      "recall:0.8695652173913043\n",
      "f1:0.9302325581395349\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import precision_score, recall_score, f1_score\n",
    "\n",
    "print(\"precision:{}\\nrecall:{}\\nf1:{}\".format(\n",
    "    precision_score(y_predicted, y_test), \n",
    "    recall_score(y_predicted, y_test), \n",
    "    f1_score(y_predicted, y_test))\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow precision of 1 and recall of 0.86! This must be a pretty awesome model, right? Nope!\n",
    "\n",
    "Remember, that a classifier that always reports 1, would have more or less the same results.\n",
    "\n",
    "We need to compare our performance against the random classifier. To do that, we use the Area Under the Receiver Operating Characteristic curve (AUC-ROC). To get a sense of what this is, we plot it below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3hU1drG4d9KQgkQSkhoISGUhN5jUEA6GKkWRBBRbFhARQREVPSIHuwdCypHVBBRKQECWBABEQQEaUovCTX0EkLKrO+PiZ4cPjSDTDKZmee+Li5mZm9m3k2Sh8Xaa7/bWGsRERHvF+DpAkRExD0U6CIiPkKBLiLiIxToIiI+QoEuIuIjgjz1wWFhYTY6OtpTHy8i4pVWr1592FobfqFtHgv06OhoVq1a5amPFxHxSsaY3X+1TVMuIiI+QoEuIuIjFOgiIj5CgS4i4iMU6CIiPiLPQDfGTDTGHDLGbPiL7cYY84YxZpsxZp0xppn7yxQRkby4MkL/CEj4m+1XAzE5vwYB71x6WSIiXiz5Z1jysvP3i9l2ifJch26tXWyMif6bXXoBH1tnH97lxpiyxpjK1tr9bqpRRMR7bF8Ik28ARzYEBEKjG6F0BABZx1II3DgNYx0QWAxuTYTIeLd9tDsuLIoAknM9T8l57f8FujFmEM5RPFFRUW74aBGRQmTrN/Dl7eDIcj53ZMHayYDBAgHkuv9EdgbsWuLWQC/Qk6LW2gnW2jhrbVx4+AWvXBUR8T5nj8PMwTC5NwSXc46+TSAEBXP65nmMariY6umTGVL8ORyBxZ3bAotC9JVuLcMdI/S9QGSu51VzXhMR8X1bvobZD8Lpg3Dlw9D2Edj/K+xaQnZUa675Kp0dqcnc3bYGD3VKIPBAc+fIPPpKt47OwT2BnggMMcZMBVoAJzR/LiI+7+xxWDDaOaUSXhf6ToYI5yK/Y6FNKFv1MgKNYXiXA1QpW5xGVcs6/1xkvNuD/A95Brox5jOgHRBmjEkBngSKAFhr3wWSgK7ANiANuC1fKhURKSy2LMgZlR+CK4dD25EQVAxrLTPX7uVfszfxSEId+sVHkdCgUoGV5coql355bLfAYLdVJCJSWJ09BvNHw69ToEI96PcZVGkKwL7jZ3lsxnq+35xK06iyxFUrV+Dleax9roiIV9k83zkqP5MKbUY4fwUVA2DW2r08NmMD2Q7LmO71uLVlNIEBpsBLVKCLiPyds8dg3ihYNxUq1IebPocqTf5nlzLBRWgSWZZx1zUkMrSEhwpVoIuI/LXN82D2UEg77Fy9cuVwCCpKVraDD5fuJDPbwZAOMbSrXYG2seEYU/Cj8twU6CIi50s7CvNHwbrPoWID6D8NKjcGYNO+kzzy1TrW7z1Bt0aVsdZijPF4mIMCXUTkf/2eBHOGQtoRaDvKubY8qCjnsrJ5a+E23lm0nbIlivB2/2Zc3aBSoQjyPyjQRUTAOSqf9wisnwYVG0L/L6Fyoz837zqcxrs/bKdnkyo80a0e5UoW9WCxF6ZAFxH5bQ7MeQjOHoV2j0LrYRBUlDPnsvhm00GuaRpB7UohfDesHVHlPXfSMy8KdBHxX2lHIWkEbPgSKjWEAdOdvwNLtqby6PT17D1+lgYRpalVIaRQhzko0EXEX/02O2dUfhzaPwatH4LAIpxIy+TZpE1MW5VCjbCSfD7oCmpVCPF0tS5RoIuIfzlzBOaNgA1fQaVGMGAmVGoAQLbDcv27y9h5+Az3tavJAx1jKF4k0MMFu06BLiL+Y1MizB2WMyp/HFoPhcAiHD2TQdngIgQGGEZcVZuIssE0iCjj6Wovmm4SLSK+78xh+OI2mDYASleBu3+AtiOwAUF8tTqF9i8tYupK5316rqpfySvDHDRCFxFft3EmzH0Y0k9Ah8ehlXNUnnIsjdEzNrB4SyrNq5Ujvnqopyu9ZAp0EfFNZw47g3zTTKjcBG6dDRXrATBjTQqPz9iABf7Vsz4DLq9GgAeaabmbAl1EfM/GGc4wP3cKOo6Blg9C4H/jLrRkMZpHh/LvaxtQtVzhXop4MRToIuI7TqdC0sOwaZazT3mvt6FiPTKzHby/aBtZ2ZYHOsbQNjacNjFhheqyfXdQoIuI97PWOSpPGp4zKn8SWj4AgUFs2HuCR75ax8Z9J+nRuEqhaqblbgp0EfFupw85p1d+S4QqzeCad6BCHdIzs3njm995b/EOypUoyrs3NyOhQWVPV5uvFOgi4p2sdV4clDQCMk5Dp6fgivv/nCvffSSN95fs4LqmETzerR5lShTxaLkFQYEuIt7n9CHnBUK/zYaI5s658gp1OHMuiwW/pnBds6rUrhTCwofbefQOQgVNgS4i3uPPUflwyEiDzk/D5YMhMIgftqQyevp69p04S6OqZahVIcSvwhwU6CLiLU4ddI7Kf58DEXFwzdsQXptjZzIYO3ct03/ZS83wknxxt/c003I3BbqIFG7WwvovnQ21MtKg81i4YjAEBP7ZTGv3kTSGtK/FkA61vKqZlrsp0EWk8Dp1AOYMg81zoWo89BoP4bEcOX2OciUCCAwwjEqoQ0S5YOpX8c7+K+6k5lwiUvhYC79+DuNbwPbvoMszcPt8bFgM01Yl0/6lRXy2cg8AXepXUpjn0AhdRAqXUwecN57YnASRLZyj8rAYko+mMXrGepZsPUx8dChX1Cjv6UoLHQW6iBQO1sK6z2HeSMg6B1f9G1rcAwGBTP8lhcdnbsAAY69pQP/4KJ9opuVuCnQR8byT+2HOUNgyHyIvzxmV1/pzc1ipYsRXD+XZaxsSUTbYg4UWbgp0EfEca+HXqTD/EcjKgKvGQYu7ybSG9xZuJdsBD3aKoU1sOG1iwz1dbaGnQBcRzzi5D2YPha0LIOoK56i8fE027D3BiC/X8dv+k/Rq8t9mWpI3BbqIFCxrYe0UmP8oZGdAwnMQfzfp2ZbX5v3O+0t2EFqyKO8NaM5V9St5ulqv4lKgG2MSgNeBQOADa+1z522PAiYBZXP2GWWtTXJzrSLi7U7ug9kPwtavIaol9HoLytcEYE/qKT5cuoPezaoyumtdv2im5W55BroxJhAYD3QGUoCVxphEa+2mXLs9Dkyz1r5jjKkHJAHR+VCviHgja2HtZJg/GhyZcPULcNldnMrIZv6qZG6IiyS2YgjfD2/nU3cQKmiujNDjgW3W2h0AxpipQC8gd6BboHTO4zLAPncWKSJe7MRemP0AbPsWqrVyjspDa/D974d4bMZ6DpxMp2lUWWpVCFGYXyJXAj0CSM71PAVocd4+TwFfG2PuB0oCnS70RsaYQcAggKioqIutVUS8ibWw5lNYMBocWXD1i3DZnRw9m8XYz9cyY81eYiqU4st7W/ptMy13c9dJ0X7AR9bal40xVwCfGGMaWGsduXey1k4AJgDExcVZN322iBQ2J1Ig8QHnZfvVWueMyquT7bD0fmcZe46m8UDHGAa3r0mxIP9tpuVurgT6XiAy1/OqOa/ldgeQAGCt/ckYUxwIAw65o0gR8RLWwi8fw4LHwDqg60sQdwepZzIp77AEBhhGd61LRLlg6lYunff7yUVxpTnXSiDGGFPdGFMU6AsknrfPHqAjgDGmLlAcSHVnoSJSyB1Phk+vc86XV2kC9/6IvexOPl+dQoeXFzHlZ2czrU71KirM80meI3RrbZYxZgiwAOeSxInW2o3GmKeBVdbaROBh4H1jzEM4T5AOtNZqSkXEH1gLv0yCBY87R+XdXobmt7PnWDqjPljBsu1HaFE9lNa1wjxdqc9zaQ49Z0150nmvjcn1eBPQyr2liUihd3yPc658x/cQfaVzrrxcNF+uTuGJmRsIDDA8e20D+l2mZloFQVeKisjFsxZWfwRfPwFY6PYKNL8NApyzuBVLF6NlzfI8c20DKpdRM62CokAXkYtzfA8k3g87FkH1ttDzTTJCInln4XYc1vJQ51iujAnnyhg10ypoCnQRcY21sGoifJMz29r9VWh+G7+mnGDkR0vZfPAU1zWNUDMtD1Kgi0jeju2GxCGwczHUaAc93+RsiQheSfqND5fupEJIcT64JY5O9Sp6ulK/pkAXkb/mcMDqifD1GDAB0ON1aHYrGEPywVNMWrabvvFRjLq6DqWLq5mWpynQReTCju2CWUNg1xKo0R56vsnJ4pWYvzqFPjnNtBaNaEcV3UGo0FCgi8j/cjhg1YfwzZM5o/I3oNktLNx8iNHTF3PoVDrNospRq0IphXkho0AXkf86utO5gmXXEqjZEXq8zpGgCjz9+Vpmrd1H7YohvDugObUqlPJ0pXIBCnQRcY7KV34A3z4JAUHQ801oOoBsCze88gPJx9J4qFMs97arSdEgVzqGiCco0EX83dEdMOt+2L0UanWCHq9zKCCMMAuBAYbHutWlarkS1K6kFreFnf6pFfFXDgeseA/eaQUH1kGv8Tj6fcHk37Pp8NIPTM5pptWxbkWFuZfQCF3EHx3Z7pwr3/0j1OoMPV5nV2ZZRn24guU7jtKyZnna6kpPr6NAF/EnDgf8PAG+fQoCi0Kvt6HJTUxbncITMxdTNDCA565ryI2XRepqTy+kQBfxF0e2O9eV71kGMV2cFwmVrgJARNlg2sSGM7ZXAyqVKe7hQuWfUqCL+DqHA1a8C9897RyVX/MO5+r34e1FO7B2M8O61KZVrTBaqV+511Ogi/iyI9th1mDY8xPEXAU9XmPN8WAeeetHthw8zfXNqqqZlg9RoIv4Ikf2f0flQcXg2vdIq3M9L3+zlYk/rqFS6eJMHBhHhzpqpuVLFOgivubwNph1HySvgNgE6P4alK7M3oOn+GT5bvq3iOKRhDqEqJmWz1Ggi/gKRzYsfwcWjoWg4nDtBE7EXMu8DQfoGw8xFUP4YUQ73UHIhynQRXzB4a0w8z5I+Rlqd4Xur/L1Hnj81cUcOZNBXHQotSqUUpj7OAW6iDdzZMNP4+H7Z52j8uve53D1njw1exNz1u2nTqUQPrg1Ts20/IQCXcRbpW5xzpWnrITa3aD7q2SXrEDvlxex73g6w7vEcnfbmhQJVIcPf6FAF/E2jmz46S1Y+CwULQHXf8jBqG6ElyxOYIDhyR71qVoumJiK6r/ib/RPt4g3Sd0MH3Zx3qg5pjOOe5fzyZnL6PjKYiav2A1A+zoVFOZ+SiN0EW+QneUclX//byhaEq7/kB0Vr2LUZxv4eedRWtcKo13tCp6uUjxMgS5S2B363TlXvnc11O0B3V7h89/SGfPGUooFBfBC70bc0LyqrvYUBbpIoZWdBcvegEXjoGgp6D0R6l8HxlC13GHa1XY206pQWs20xEmBLlIYHfrNua583y9QtyfnEl7kzeUnYO8Whl+lZlpyYQp0kcIkOwuWvQ6LnoNiIXDDR6wu1ZaRH6xje+oZ+sSpmZb8NQW6SGFxcJNzrnzfGqh3DWc6PceLS48y6aefqFImmEm3x9M2VncRkr/m0rJFY0yCMWazMWabMWbUX+zTxxizyRiz0Rgzxb1liviw7CxY/BJMaAvHk+GGj6DPJPZllmLKz3u45fJqLHiojcJc8pTnCN0YEwiMBzoDKcBKY0yitXZTrn1igEeBVtbaY8YYrZ8SccXBjc658v1rof61nGw/jjnbM7kJZzOtJSPbU1EnPcVFrky5xAPbrLU7AIwxU4FewKZc+9wFjLfWHgOw1h5yd6EiPiU7E5a+Bj88D8XLwA2TmG8v54n3NnD0TAYtaoRSM7yUwlwuiitTLhFAcq7nKTmv5RYLxBpjfjTGLDfGJFzojYwxg4wxq4wxq1JTU/9ZxSLe7sAG+KAjfP8M1OtJ6q2LuW9tJPd8uprwUsWYNbgVNcPVTEsunrtOigYBMUA7oCqw2BjT0Fp7PPdO1toJwASAuLg466bPFvEO2Zmw9FX44QUILgt9PiG7Tg9ueHkR+06kM+Kq2gxqU0PNtOQfcyXQ9wKRuZ5XzXkttxRghbU2E9hpjNmCM+BXuqVKEW93YL1zrvzAOmjQmwOt/0WFChHOZlo96xNZroRa3Molc2UosBKIMcZUN8YUBfoCieftMxPn6BxjTBjOKZgdbqxTxDtlZ8Ki52FCOzh1AEefT/io8uN0eHsDn/7RTKt2BYW5uEWeI3RrbZYxZgiwAAgEJlprNxpjngZWWWsTc7Z1McZsArKBEdbaI/lZuEiht3+dc135gfXQ8AZ2XDaGkXNTWLV7E21iw+lQR4vBxL2MtZ6Zyo6Li7OrVq3yyGeL5KusDFjyMix5CYJDofurTD3ViDGJGwkuEsiY7vW4rlmErvaUf8QYs9paG3ehbbpSVMSd9q9zzpUfXA+NboSE56BEKFHbD9OpbgX+1bMB4SHFPF2l+CgFuog7ZGU4R+RLXoYS5cm4YTKvJdeCxYcYmRBKy5phtKypZlqSv7Q+SuRS7f8V3m/vvEioQW/W9JhPwvxSvL1oO0fPZOCpaU3xPxqhi/xTWRmw+EXnqLxkOGd7T+a57dF8/NFvRJQN5uPb42mj/itSgBToIv/EvjUwczAc2giN+0HCOPaeCmLq1KXcekU0I66qTcli+vGSgqXvOJGLkXXOeaXn0lehVAVOXzeZGWkNGRBcjlrBsGRke91BSDxGgS7iqr2/wKzBcGgTtslNfBP5IKMTkzmetpGWNctTM7yUwlw8SoEukpesc84Tnktfg1IVOH7NZB5ZX4kFX2ynYUQZPr69hZppSaGgQBf5O3t/ca4rT/0NmtxMdpdn6DV+LQdOpPLo1XW4o3V1gtRMSwoJBbrIhWSdc97X88fXoVRFjvSaTLnG3QgMMDzdqwGR5YKpoVG5FDIaWoicL2U1vNcGlr6Co3E/Pm3+Oa2nB/3ZTKttbLjCXAoljdBF/pCZDovGwbI3IKQy+7p/wpCfy/PL8mTa1Q6nY92Knq5Q5G8p0EUAUlY558oPb4Zmt/BF6D08NmMPJYud4dUbG3NNEzXTksJPgS7+LTMdFv0blr0JIVXg5q+gVicith+mS/1zPNWzPmGl1ExLvIMCXfxX8kpnv/LDW8hqcgtvBN5CxtYQRtVCzbTEK+mkqPifzLPw9RMwsQtkpPF7p0l03nY9b/x4iFPpmWqmJV5LI3TxL8k/O+fKj2wlo8ktPJ/dnw/nHCEq1DLlzha0rKVRuXgvBbr4h8yzsPAZ+Gk8lKkKA2ayJ+QyJr+5hDtbV2dYl1hKFNWPg3g3fQeL79uzwjlXfmQb6Y0HMiNsEP1q1qcWsGRkB91BSHyGAl18V0YafP8s/DQeW6Yqy1tNZMjy0pxM302LOtWoEV5KYS4+RYEuvmnPcudc+dHtpDUeyIjj1zP3u1M0qhrM5N4tdKWn+CQFuviWjDTnXPnyt6FsJNkDZnH1dMuBE2d4rGtdbmsVrWZa4rMU6OI7dv/knCs/uoPTjQYS3PUZAouHMLZXKlGhJYgOK+npCkXylYYq4v0y0mDeKPjP1VhHNnObTiBuTQKf/nIEgDax4Qpz8QsaoYt3273MOVd+bCfHGgzkngM9WPHTOTrWCaNLfTXTEv+iQBfvlHEGvnsaVrwH5arxzWUfct+yEoQUt7zetwk9G1dRMy3xOwp08T67fnTe2/PYTmz8IEynpwhJSafrqT2M6V6P8mqmJX5KgS7eI+MMfPsv+Pk9HGWjmVx7PCmmOY8WLcnlNUpyeY3ynq5QxKMU6OIddi6BxCFwbBf76wxk4J4ENv/qYMDl2VhrNb0iggJdCrtzp+Hbp2Dl+zjKVuf9Gm8xbm0o1coXZ8pdDdXiViQXBboUXjsXw6whcHwPXH4fOxsO47V3VzOoTTUe6hRLcNFAT1coUqi4tA7dGJNgjNlsjNlmjBn1N/tdb4yxxpg495UofufcaZj7MEzqQTYBJMVNhIRx1IwIZ+kj7Rndta7CXOQC8hyhG2MCgfFAZyAFWGmMSbTWbjpvvxDgQWBFfhQqfmLHD5A4BHs8me01b2XAjs4c/imQOi1OUyO8lFawiPwNV0bo8cA2a+0Oa20GMBXodYH9xgLPA+lurE/8xblTMGcYfNyTLIIYV+lVOm28iophocx94Eo10xJxgStz6BFAcq7nKUCL3DsYY5oBkdbaucaYEX/1RsaYQcAggKioqIuvVnzTjkUw6344kYzj8sFc/WsbUo7AE91rM7BlNIEBWsEi4opLPilqjAkAXgEG5rWvtXYCMAEgLi5ON270d+dOOe/tufo/ZJatScBt8wmsdjlP1jxMVGgJosqX8HSFIl7FlSmXvUBkrudVc177QwjQAFhkjNkFXA4k6sSo/K3t38PbV2BXf8SvUQNofvhJPklx9l5pHROmMBf5B1wJ9JVAjDGmujGmKNAXSPxjo7X2hLU2zFobba2NBpYDPa21q/KlYvFu6Sdh9oPwyTWcM0UZWeZFem25mhaxEVzdsLKnqxPxanlOuVhrs4wxQ4AFQCAw0Vq70RjzNLDKWpv49+8gkmP7Qkh8AE7uZWP0QPpsaU/x4JK8dVN9ujWsrKs9RS6RS3Po1tokIOm818b8xb7tLr0s8SnpJ+Hrx+GXSdiwWMztX3M6swZdgpN5ons9QksW9XSFIj5BV4pK/tr2LSQ+iD21j2UVb2Zp5F08EtmEFkALNdMScSvdsUjyR/oJ52X7n15PminOPUXH0X93V846imCtFjiJ5AeN0MX9tn4Lsx/AntrPorD+3JPSmSph5Zh2dyPiq4d6ujoRn6VAF/c5exy+fgzWfArhddjb5T0GTzvDbW2jGdophuJF1H9FJD8p0MU9tn4DiQ9gTx9gXfTtNO4/jqpFirO0eoZOeooUEM2hy6U5exxmDobJvTlJCW7mWW7Y2oWdJ7IBFOYiBUgjdPnntnztnCs/fYg5pfvx8KEEGkSFk9S7EdXDSnq6OhG/o0CXi3f2GMwfDb9OwYbX5a6MYSw7HsXoHrUZcIWaaYl4igJdLs6WBTD7QezpQ9grhxPQdiS37TrFk6EliAxV/xURT9Icurjm7DGYcQ9M6cMRR0muyxzLpOI3Q1AxWtUKU5iLFAIaoUveNs+D2UOxZ1KZGtyXMUe60qF+BN3UTEukUFGgy19LOwrzH4V1UzlaKpbbMh5gb2Bt3uhfX50RRQohBbpc2O9JMGcoNu0Ipu0jbKt6G7XWHGJS97qULaGliCKFkQJd/lfaUZj3CKyfxoHgGObUeZ47219LPBAfo1G5SGGmQJf/+n0uzB6KI+0oE4P68sLxrtxUvBbWWvUqF/ECCnTJGZWPhPVfsLd4Le5KH0p6WH0m39yIy6LVTEvEWyjQ/d1vc2DOQ3D2KEfjh9N1eVP6t63FAx3VTEvE2yjQ/dWZI85R+YYvORxSh7BB0wmt1JBFbTMop/4rIl5JFxb5o02J2Ldb4Ng4k/HcSJtjj7MzqAaAwlzEi2mE7k/OHIGk4bBxOruK1OLe9IcpFdWYxOvVTEvEFyjQ/cWmWTBnGDb9BO8H3cT4c90Y3rMB/VtUI0DNtER8ggLd1505nDMqn4Gt3BhzayINTldkbmgJqpZT/xURX6JA92UbZ2LnPozj7HFez76R0HrDGVgxlpYVPV2YiOQHBbovOp3qHJVvmsnWwBiGpI8gpkE8A5pU83RlIpKPFOi+ZsN0SBpO9tmTvJJ1I18FXc9T/RuT0KCSpysTkXymQPcVp1Mh6WHnyc8qzdjUaRyHtgezoFs9ypQo4unqRKQAKNC9nbWwcTp27nCy00/xQ8R9dLx9LA0Dg3ixmaeLE5GCpAuLvNnpQzBtAHx5O5vSy3P1uWdZVnkANkCX7Iv4I43QvZG1sOErHEkjyE4/zUuZ/fg+tA/PDWxK82rlPF2diHiIAt3bnDoIc4fB73PIqNiUPmkDaNe2NbM71KJYkEbmIv7MpUA3xiQArwOBwAfW2ufO2z4MuBPIAlKB2621u91cq3+zFtZ/iSNpBI6MMwR2epriLYfwSbpDJz1FBHBhDt0YEwiMB64G6gH9jDH1ztttDRBnrW0EfAm84O5C/dqpg9jP+8P0O1mfHkaPjHHsqnMnBAQqzEXkT66M0OOBbdbaHQDGmKlAL2DTHztYa7/Ptf9y4GZ3Fum3rIX1X5A9dwTZ59J4IbM/6yNvYvz1TdRMS0T+H1cCPQJIzvU8BWjxN/vfAcy70AZjzCBgEEBUVJSLJfqpUwecN57YnMSmgNo8au+lb48OjI6PUjMtEbkgt54UNcbcDMQBbS+03Vo7AZgAEBcXZ9352T7DWlg3jeykkQRkp2O6PMvp8D5MCA+hStlgT1cnIoWYK4G+F4jM9bxqzmv/wxjTCXgMaGutPeee8vzMyf04Zg8lYOt81jpi2dnqRXq3bMcVnq5LRLyCK4G+EogxxlTHGeR9gZty72CMaQq8ByRYaw+5vUpfZy38OpWspJFkZ6TzQubNpNa/jTGtGnq6MhHxInkGurU2yxgzBFiAc9niRGvtRmPM08Aqa20i8CJQCvjCGAOwx1rbMx/r9h0n98OcobDFOSp/rtj93N3nKjrXU49bEbk4Ls2hW2uTgKTzXhuT63EnN9fl+6yFXz/Dzh+FycpgT/wYpp/twIddG1AmWEsRReTi6UpRTzi5j6xZ9xO0/VuSSzUm6q7/EFW+Jv/2dF0i4tUU6AXJWlg7hcykR8jOPMezWbdQpO49PBpaAy1EFJFLpUAvKCf2kjHzforu/I5fHHV4p/RQHuyTQNMoNdMSEfdQoOc3a2HNp7BgNEHZmYyzt1G89T1M6BBL0SB1LxYR91Gg56cTKZybcT/Fdi3EVmtFQK+3uC84Uic9RSRfaIiYH6zFrp5ExpstyN75I087bmN3988htIbCXETyjUbo7nY8mbPTBxO85wdWZ9fjs8ojGNbnKqLVTEtE8pkC3V2shV8+xi54DJuRyVh7BzW7PcBr8dXUTEtECoQC3R2OJ5P21X2USF6Mib6SzU2f4c7o2lQuo2ZaIlJwFOiXwlqyVv6H7AWPY7OyWFZ3NC37jKBpgE5NiEjBU6D/U8f3cGravYTsW8qK7Pp8G/M49/foBApzEfEQBfrFshZW/4eMeY9hshw8F3Q3l/UZxpP1Knm6MhHxcwr0i3FsNzbxfszOH0iv3Ip3yw7lvl7tKV1cSxFFxPMU6K5wODi74gMCvhkDGIp1f43SzQcy0mj1iogUHgr0vCDeUUoAAAdwSURBVBzbxdHP7ib00HKWOBqyrtnT3Ne8PUZhLiKFjAL9rzgcnP7xPYosfIoiDsNrJYbQvt9wBquZlogUUgr0Czm2C2YNodSuJfxIYzZf9iz3JbRSMy0RKdQU6Lk5HBxf8g4lfxhLUJEimJ5v0rBuX1oFF/V0ZSIieVKg53Ac2Unqp3dS8dgqltjGRA/4gMjqsZT2dGEiIi5SoDscHF40nlJLxhLsCGBC6DAS+g8nUs20RMTL+HegH92BnTmYsD3LWEoTjnV8ibuujNMKFhHxSv4Z6A4HqQvfIGzF85iAIuxo+QIxLW6hopppiYgX87tAP3doK6mf3kXVk2tILt+ayFsmUKNMhKfLEhG5ZP6zDs/hIHney9i3W1L6xGamVH6UUrdNB4W5iPgI/xihH9nOgY/vIPLEGn4MaA49XuOmpo08XZWIiFv5dqA7snEsf5eAhWMJCyjCl1GPcVW/oYRoXbmI+CCfDfRTe3/j8OS7qJ62HmITCOr+Gr1LV/Z0WSIi+cb35tAd2fw+YxxF3m9D6JkdzKn5JLbvZ6AwFxEf51Mj9GN7NnJ0yl3USd/I8iLxlO0znu4xsZ4uS0SkQPhGoDuyYfnblPnuGQKyA/mmzlja9R5MkaBAT1cmIlJgvD7QD+5Yh51xH5VOrSegdleCurxI5/JVPV2WiEiBc2kO3RiTYIzZbIzZZowZdYHtxYwxn+dsX2GMiXZ3oedzZGWxesq/KDupA8VP7iS181vQdwolFeYi4qfyHKEbYwKB8UBnIAVYaYxJtNZuyrXbHcAxa20tY0xf4HngxvwomOSfOb5qGic3fE3z7N2sKtGSyje9Q0RkdL58nIiIt3BlyiUe2Gat3QFgjJkK9AJyB3ov4Kmcx18CbxljjLXWurFWSP4Z+5+ulHFkUgbYEDuY5n2fwQT43mIdEZGL5UoSRgDJuZ6n5Lx2wX2stVnACaD8+W9kjBlkjFlljFmVmpp68dXuWoKx2RgAE0iDyPIKcxGRHAWahtbaCdbaOGttXHh4+MW/QfSVEFgMTCAmsKjzuYiIAK5NuewFInM9r5rz2oX2STHGBAFlgCNuqTC3yHi4NRF2LXGGeWS82z9CRMRbuRLoK4EYY0x1nMHdF7jpvH0SgVuBn4DewEK3z5//ITJeQS4icgF5Brq1NssYMwRYAAQCE621G40xTwOrrLWJwIfAJ8aYbcBRnKEvIiIFyKULi6y1SUDSea+NyfU4HbjBvaWJiMjF0BIREREfoUAXEfERCnQRER+hQBcR8REmv1YX5vnBxqQCu//hHw8DDruxHG+gY/YPOmb/cCnHXM1ae8ErMz0W6JfCGLPKWhvn6ToKko7ZP+iY/UN+HbOmXEREfIQCXUTER3hroE/wdAEeoGP2Dzpm/5Avx+yVc+giIvL/eesIXUREzqNAFxHxEYU60AvjzanzmwvHPMwYs8kYs84Y850xppon6nSnvI45137XG2OsMcbrl7i5cszGmD45X+uNxpgpBV2ju7nwvR1ljPneGLMm5/u7qyfqdBdjzERjzCFjzIa/2G6MMW/k/H2sM8Y0u+QPtdYWyl84W/VuB2oARYFfgXrn7XMf8G7O477A556uuwCOuT1QIufxvf5wzDn7hQCLgeVAnKfrLoCvcwywBiiX87yCp+sugGOeANyb87gesMvTdV/iMbcBmgEb/mJ7V2AeYIDLgRWX+pmFeYT+582prbUZwB83p86tFzAp5/GXQEdjjCnAGt0tz2O21n5vrU3Leboc5x2kvJkrX2eAscDzQHpBFpdPXDnmu4Dx1tpjANbaQwVco7u5cswWKJ3zuAywrwDrcztr7WKc94f4K72Aj63TcqCsMabypXxmYQ50t92c2ou4csy53YHzX3hvlucx5/xXNNJaO7cgC8tHrnydY4FYY8yPxpjlxpiEAqsuf7hyzE8BNxtjUnDef+H+ginNYy725z1PLt3gQgofY8zNQBzQ1tO15CdjTADwCjDQw6UUtCCc0y7tcP4vbLExpqG19rhHq8pf/YCPrLUvG2OuwHkXtAbWWoenC/MWhXmEfjE3pyZfb05dcFw5ZowxnYDHgJ7W2nMFVFt+yeuYQ4AGwCJjzC6cc42JXn5i1JWvcwqQaK3NtNbuBLbgDHhv5cox3wFMA7DW/gQUx9nEyle59PN+MQpzoP95c2pjTFGcJz0Tz9vnj5tTQ37fnLpg5HnMxpimwHs4w9zb51Uhj2O21p6w1oZZa6OttdE4zxv0tNau8ky5buHK9/ZMnKNzjDFhOKdgdhRkkW7myjHvAToCGGPq4gz01AKtsmAlArfkrHa5HDhhrd1/Se/o6TPBeZwl7opzZLIdeCzntadx/kCD8wv+BbAN+Bmo4emaC+CYvwUOAmtzfiV6uub8Pubz9l2El69ycfHrbHBONW0C1gN9PV1zARxzPeBHnCtg1gJdPF3zJR7vZ8B+IBPn/7juAO4B7sn1NR6f8/ex3h3f17r0X0TERxTmKRcREbkICnQRER+hQBcR8REKdBERH6FAFxHxEQp0EREfoUAXEfER/weyggh8PT8rDAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import roc_auc_score, roc_curve\n",
    "from matplotlib import pyplot\n",
    "\n",
    "blr_fpr, blr_tpr, thresholds = roc_curve(y_test, y_predicted)\n",
    "\n",
    "pyplot.plot([0, 1], [0, 1], linestyle='--')\n",
    "pyplot.plot(blr_fpr, blr_tpr, marker='.')\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5116279069767442"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(y_test, y_predicted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, our score is abysmal. \n",
    "\n",
    "Let's try a more sophisticated approach, called Random Forests. What this classifier does is that it trains multiple decision trees on random permutations of the data, and, at prediction time, takes a majority vote by consulting all those trees."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "rf = RandomForestClassifier(n_estimators=50).\\\n",
    "  fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "commits_on_files_touched: 0.3109899464414616\n",
      "test_lines_per_kloc: 0.11138705634975352\n",
      "requester_succ_rate: 0.09884664830471104\n",
      "prev_pullreqs: 0.09599520441916874\n",
      "files_changed: 0.08258794530288405\n",
      "perc_external_contribs: 0.07959674581191754\n",
      "num_comments: 0.07136638884907644\n",
      "num_participants: 0.056678922609242524\n",
      "team_size: 0.05460291004796252\n",
      "num_commits: 0.03794823186382197\n"
     ]
    }
   ],
   "source": [
    "for feature, coef in sorted(zip(columns, rf.feature_importances_), key=lambda x: x[1], reverse=True):\n",
    "    print(\"{}: {}\".format(feature, coef))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_predicted = rf.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1,\n",
       "       1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 0,\n",
       "       1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1,\n",
       "       0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1])"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_predicted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks more normal; we can also see that our recall has improved significantly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "precision:0.9535714285714286\n",
      "recall:0.9270833333333334\n",
      "f1:0.9401408450704226\n"
     ]
    }
   ],
   "source": [
    "print(\"precision:{}\\nrecall:{}\\nf1:{}\".format(\n",
    "    precision_score(y_predicted, y_test), \n",
    "    recall_score(y_predicted, y_test), \n",
    "    f1_score(y_predicted, y_test))\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that this classifier does much better than binary logistic regression in terms of AUC score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd1yV5f/H8dfFVBBRFBeKOMCdi3DvkZqpqZk5srTM9tJsiANyZGpllmVmZdk3Z64clTlRQNx7D3AiKigbzvX741A/MgfKgZtzzuf5ePSIc+7bc39uwbeX933dn0tprRFCCGH9HIwuQAghhGVIoAshhI2QQBdCCBshgS6EEDZCAl0IIWyEk1EHLlmypPbz8zPq8EIIYZV27NhxRWvtfbtthgW6n58fUVFRRh1eCCGsklLqzJ22ySUXIYSwERLoQghhIyTQhRDCRkigCyGEjZBAF0IIG3HPQFdKzVFKXVZK7b/DdqWUmq6UOq6U2quUamD5MoUQouCbs38OkRci//Ve5IVI5uyfc9dtlpKTEfr3QKe7bO8M+Gf9NxSYmfuyhBDC+tQuUZvhG4f/E9yRFyIZvnE4tUvU/v9te+fC5qlE7p37zzZLuec8dK31JqWU31126Q7M1eY+vOFKqWJKqbJa6wsWqlEIIaxCUNkgQhoF88q6l6lRtBIH40/QvsRDRO77ETJTaaoL8fKOj+iSmMR6NzemBI4kqGyQxY5viQeLfIDobK9jst77T6ArpYZiHsXj6+trgUMLIYQFpadAagKkJEBqfNb/E/79/5T4rK/j/7Nts0rhQ6+iJDs5sfPaIdCa3y5v/9chtFIs8SjCC9cTCIqPtWj5+fqkqNZ6FjALIDAwUFbWEEJYTkZqtnCNv23g3jGs/943M+3ex3EpAq5FoZAnFCoK7t4keFXk44yLLE27QDlHdzx0Bv28H2ZB3C5CA4aw+0gJFuxPIMD7ACeKr6LPjRss8ChCkKc3lhufWybQzwEVsr0un/WeEELkzK1hfNsgjr/7yDkz9d7H+SeMi5r/71YCilf6/3D+O6iz73Pr/x0c//WRm2I2MW7bOOLS43i00qOEnQ/j01bTCSobROD5CIaufYOkmKfo2qQsO5IjmFpzJEHxsQR5ejP84GymeFe32GUXSwT6cuAVpdQvQCMgXq6fC2FHMtJuCeI7BO5tgzjeAmGcPXCzgvlOoXxLGOdGQloCkyMns+zEMqoWq8r0NtOJuBhBT/+e+Beth9aaxuUaMbT6GOIDTlDW8wb9Skz5J7yDgCne1dkft99iga7utaaoUup/QGugJHAJGAM4A2itv1JKKWAG5pkwScCzWut7dt0KDAzU0pxLCIP9J4xvF8QJkHL9ztsyUu59HGf3f4frbQPX886jYtei4GhYL8H/2BSziXFbxxGXEsfg2oMZVncYLo4uaK1Zuvsc41YcZGSn6jwVZPl7hUqpHVrrwNtty8ksl6fusV0DLz9gbUKIB5WZ/u+R790uVdzp/QcJ48LFoXjFW4LYesI4N+JT45m8fTLLTyw3j8rbTadWiVoAnL+ezAe/7mP9kVjq+xYjsGLxfK/PNn6XhbA2t4bxHa8Z32XknJF87+M4u/37Bl6hYlDM996j4r9H0DYUxrm1MXoj47aN42rKVYY+NJQXHnoBF0cXAJbtPscHv+4n06QZ3bUmg5r64eig8r1G+U4JER0JpzeDXwuokINrmZkZ/x35/id47xLEKfH3GcbZbtZ5VsgWuHcZFRfyBFcPcHTO/e+PnYtPjeejyI9YcXIF/sX9mdFuBjVL1PzXPp6FnalXoRgTe9ahgpebQZVKoAt7Fx0JP3Q1X0t2cIJ6/aGQx11u7CVAetK9P/c/YVwUPMvfZSbFrbMsJIwLgg3RGwjZFsK1lGsMqzuMoXWG4uzoTEamiW+3nCI908Qrbf1pXa0UrQK8Md9SNI4EurBvB5eZp8wBmNJh5/fgVPi/I967hvFtbuxJGFu1+NR4JkVOYuXJlQQUD+CLdl9Qo0QNAA6eT2Dk4r3sOxfPow+VRWuNUsrwMAcJdGHPUm/CoRXmr5UDOLrAwF+hYlNj6xKGWn92PSHhIVxPuc6LdV/k+TrP4+zoTGpGJjP+Os7MDSco5ubMl/0b0Ll2mQIR5H+TQBf2SWtY/grER0OnSebLKDm9hi5s0vWU60zaPonfTv5GteLVmNl+JtW9qv+z/fSVJL7aeIJu9coR/GhNiru7GFjt7UmgC/u0bQYc+BXaj4PGLxpdjTDYurPrCN0WSnxqPC/VfYnn6jyHs6MziakZ/HHwEj3q+1CtjAfr3mqNbwnjbnreiwS6sD8nN8Ifo6FGN2j2utHVCANdT7nOhMgJrD61mupe1fm6w9dU86oGwOZjsby3ZB/nridT26coVUt5FOgwBwl0YW/iY2DRs1DCH3p8CQXo+qfIX+vOrCMkPISEtARervcyQ+oMwdnBmfikdMavOsiCqBgql3Rn/tAmVC3lYXS5OSKBLuxHegrMH2ieoth3nnlqoLA711KuMTFiIqtPr6aGVw1mdZj1z6g806Tp9dVWTl1J5KXWVXitnT+FnC3X/yWvSaAL+7H6HTi/E56cByX9ja5GGODPM38SGh5KQloCr9R7hcF1BuPs4MzVxDSKFXbG0UEx4pFq+BQrTG0fT6PLvW8S6MI+7Pgedv4ALYZDja5GVyPy2dWUq0yMmMia02uo4VWDbzp+Q0DxALTWLN4RQ8hKczOtfo18eaRWGaPLfWAS6ML2xeyAVSOgSjto877R1Yh89vvp3xkfMZ6EtARerf8qz9Z+FmcHZ2KuJfH+r/vZdDSWhhWLE1TJy+hSc00CXdi2m7GwYCB4lIFesy3aD1sUbFdTrjI+fDy/n/mdmiVqMrvjbPyLmy+1/borhlG/7kcD47rVYmDjijgY0EzL0iTQhe3KzDDPaEmKgyG/g5v1j8BEzqw9vZbx4eO5mX6T1xu8zjO1nsHJ4f/jzsvdlYZ+Xkx4vDblixfsqYj3QwJd2K4/x5i7KD7+NZSta3Q1Ih/EJccxPmI8f5z5g1olahHaLBT/4v6kZ5r4csNxMjI1r7Xzp1WANy39Sxaox/YtQQJd2Kb9i81PgwYNhbp9ja5G5DGtNWvPrGVC+IT/jMr3n4tn5OK9HDifwGN1yxWoZlqWJoEubM+lg7DsFajQGDqON7oakceuJF9hQsQE/jjzB7VL1ObD5h9SpVgVUtIzmfb7Yb7edJLibi58NaABnWqXNbrcPCWBLmxL8nWYP8D80FCfH8Cp4DVQEpahtWbN6TVMiJhAYnoibzR4g0G1Bv1zrfxMXBLfbD5Jz/o+jHq0Jp5utt/SWAJd2A6TCX4dBtfPwDO/mWe2CJt0JfkK48PH8+fZP6lTsg6hzUKpUqwKiakZLD8QQ88G5alWxoO/3m5t6ApC+U0CXdiOzVPg6Gro/DH4Nja6GpEHtNasPrWaCZETSE5P5q2GbzGw5kCcHJzYeDSW95fs43x8Mg+V96RqKQ+7CnOQQBe24ujvsH4CPNQXgp43uhqRB64kXyF0Wyh/Rf/FQyUfIrRZKJWLVeZaYhqhv+1myc5zVPF2Z+EL1tNMy9Ik0IX1u3oSljwHZWpD10+kg6KN0Vqz6tQqJkZOJDk9mbcbvs3AmgNxdHD8p5nWmbgkXmlTlVfaVrWqZlqWJoEurFtaIvwyAFDw5E/gYl//xLZ1sUmxhIaHsj56PXW96xLSLITKnpWJu5lKcTcHHB0U73aqjk/xwtQqZ33NtCxNAl1YL61hxetw+SAMWATF/YyuSFiI1pqVJ1cyKXISqZmpDA8czoAaA3BQDiyIiubDlQcZ2bk6/RtVpKMVN9OyNAl0Yb0ivoJ9C6FtMFRtb3Q1wkJik2IJCQ9hQ/QG6nnXI6RZCJU8KxF9NYn3f93H5mNXCPLzoknlEkaXWuBIoAvrdDoMfh8F1btC87eMrkZYwN+j8omRE0nLTGNE4Aj61+iPo4MjS3bGMGrpfhQQ2qM2/YN8baKZlqVJoAvrk3AeFj5jvsTSYyY4OBhdkcily0mXCdkWwsaYjdQvVZ+QpiH4efr9s71kEVeCKnkx/vE6+BQrbFyhBZwEurAuGamw4GnzzdBBK6BQUaMrErmgtWbFyRVMipxEemY67zz8Dv2q98OkFTP+OkamCV5v70/LAG9aBngbXW6BJ4EurMua9yBmOzzxA5SqbnQ1IhcuJV4iJDyETTGbaFCqASHNQqhYtCL7z8UzYtFeDl1IoHu9/2+mJe5NAl1Yj10/QdS30Ox1qNXD6GrEA9Jas+zEMiZHTibdlM7Ih0fSr0Y/0jI0k1Yf5pvNJ/Fyd+HrgQ2tejk4I+Qo0JVSnYDPAEdgttZ60i3bfYEfgGJZ+7yrtV5l4VqFPTu/C1a+BZVaQdvRRlcjHtClxEuM2zaOzec206BUA0KbheJb1BeAs1dv8O2Wk/RuUJ73u9Swi2Zalqa01nffQSlH4CjQAYgBtgNPaa0PZttnFrBLaz1TKVUTWKW19rvb5wYGBuqoqKhcli/sQmIczGpl/nroBnAvaWQ14gForVl6fCkfb/+YDJ3B6w1e56nqT5GYmsma/Rd5IrACADHXkmxqBaG8oJTaobUOvN22nIzQg4DjWuuTWR/2C9AdOJhtHw38fXfKEzj/4OUKkY0pExYPhpuXYfAaCXMrdDHxImO3jSXsXBgNSzcktGkoFYpWYP3hy3zw6z4uJqRQ37cYVUt5SJjnUk4C3QeIzvY6Bmh0yz5jgd+VUq8C7sBtn/JQSg0FhgL4+vreb63CHv0VCic3QPcvwKeB0dWI+/D3qHzy9slk6kzeC3qPvtX7cj0pgzfn7+bXXefwL1WERS82tdtmWpZmqZuiTwHfa62nKqWaAD8qpWprrU3Zd9JazwJmgfmSi4WOLWzVwWWw5RMIHAz1BxhdjbgPFxMvMnbrWMLOhxFYOpCQZiFU8KhApknTe+ZWzl5N4rV2/rzcpgquTvbbTMvSchLo54AK2V6Xz3ovuyFAJwCt9TalVCGgJHDZEkUKOxR7BJa+BD6B0GnSvfcXBYLWmiXHlvBx1MeYtIn3G73Pk9WeJO5mOiaTxtFB8X6XGvgUL0yNsvIMgaXl5BG77YC/UqqSUsoF6Assv2Wfs0A7AKVUDaAQEGvJQoUdSUmAX/qDc2HoMxecXI2uSOTAhZsXGPbnMMZuG0vNEjVZ3G0xfav1ZWFUDG2nbuDnyLMAtK9ZWsI8j9xzhK61zlBKvQKsxTwlcY7W+oBSKgSI0lovB94GvlFKvYn5Bukz+l7TZ4S4HZMJlr5o7nE+aDl4+hhdkbgHrTWLjy1mStQUTNrEqEajeKLaE8RcTaH/7Ai2noijUSUvmleVG9p5LUfX0LPmlK+65b3R2b4+CDSzbGnCLoV9AodXwiMTwa+50dWIezh/8zxjt45l24VtBJUJYlzTcZT3KM+iHTEEL92Po4Ni/OO1eephaaaVH+RJUVFwHF8H60Khdm9o/KLR1Yi70Fqz6NgipkZNRWtNcONgegf0xkGZr+KWLupK0yol+PDx2pT1lGZa+UUCXRQM107D4iFQqiZ0my7LyBVg52+eZ8zWMYRfCKdR2UaMazoO70Jl+XzdCUxa82aHAFr4e9PCX5pp5TcJdGG89GSYPwC0Cfr+BC7uRlckbkNrzcKjC5kaNRWA4MbBPBHwBHtj4hm8aAtHLt2gZ30faaZlIAl0YSytYeWbcHE/9FsAXpWNrkjcxrmb5xgTNoaIixE0LtuYcU3HUdylNBNWHeLbLaco5VGI2U8H0r5maaNLtWsS6MJY22fDnv9B6/cgoKPR1YhbmLSJhUcWMnXHVByUA2OajKGXfy+UUhy9dIMftp6hb5Av73auTtFC0kzLaBLowjhnI2DNuxDQCVq+Y3Q14hYxN2IYs3UMkRcjaVK2CeOajsPdqSQLd8TQJ7ACAaU92DCiNeVkBaECQwJdGOPGRfPKQ54V4PGvZRm5AsSkTSw4soBpO6bhoBwY22QsPf17sv7IZd5fsonLN1Jo4FucqqWKSJgXMBLoIv9lpMGCQZCaAAOXQOFiRlckskTfiGbM1jFsv7idZuWaMabJGFzw4o35u1m2+zzVSnvw1cCGVC1VxOhSxW1IoIv89/soiA6HXt9C6VpGVyMwj8p/OfwLn+78FEflyLim43i86uOYNHSYtpHoa0m82T6AF1tXwcVJ/jVVUEmgi/y15xeI/BqavAJ1ehtdjQCiE6IZvXU0UZeiaObTjLFNxuJgKobW4Oig+ODRGpQv7ka1MtLitqCTv2pF/rmwF1a8Dn4toP04o6uxeyZtYt6hefRa0YvDVw8T0jSEL9p8ybr9qbSdspF5Wc202tUoLWFuJWSELvJH0lXzw0OFvaD3d+AoP3pGOptwltFbR7Pj0g6a+zRnTJMxpCR70P/bCMJPXqVplRK0kic9rY78qRJ5z5QJi5+DGxfg2dVQRILCKCZt4n+H/8enOz7F2cGZ0GahdK/SnYU7YgheugkXRwcm9azDkw9XkKc9rZAEush7GybCiXXQ9VMof9u1bUU+OJtwluCwYHZe3kkLnxaMaTKG0u7mJzt9ihWmZYA3od1rU8azkMGVigclgS7y1uHfYNPHUH8gNHzG6Grs0t/XyqfvnI6zgzMfNvuQRyo+yswNJ9H6Om91rEazqiVpJv3KrZ4Eusg7V47BkhegXH3oMkU6KBrgTMIZRoeNZuflnbQs35LRjUdzPs6FbjPCOHrpJr0alJdmWjZEAl3kjdQb5pugTi7Q50dwln/G56dMU6Z5VL5rOi6OLkxoPoG25Tsx7Y9jzAk7RZmihZjzTCBtq0szLVsigS4sT2tY9jJcOQoDl0KxCvf+NcJiTsefJjgsmN2xu2lVvhWjm4ymlFspjl26wY/hZ+jfyJeRnarjIc20bI4EurC8rdPh4DLoEAqVWxldjd3INGXy06Gf+HzX57g6ujKh+QRalH2ENfsv0jcI/Et7sHFEa1lByIZJoAvLOrkR/hwLNXtA01eNrsZunIo/RXBYMHti99C6QmtGNx7NrlOZdPxkE3GJaQT6eVG1VBEJcxsngS4s53o0LHoWSgZA9y/kJmg+yDRl8uPBH5mxewaujq5MbDGRRt7tGbf0ICv3XqB6GQ9mDwqUZlp2QgJdWEZ6CiwYCJnp8ORP4CoBktdOxp8kOCyYvbF7aVOhDaObjKa4awnaTd3A+espDO8YwAutquDsKB0+7IUEusg9rWHV23B+F/T9GUr6G12RTcs0ZTL34Fxm7JpBYefCfNTiIxqUaIOXayEcHBRjHqtF+eKF8S8t/VfsjfzVLXJvx/ew6ydoOQKqP2p0NTbt5PWTPL36aabtmEZzn+YseexX4i7Xov20TcyLOANAm+qlJMztlIzQRe5Eb4dVI6Bqe/O6oCJPZJgymHtwLl/s+gI3Zzcmt5xMgHtzXvlpP5GnrtK8aklaVytldJnCYBLo4sHdvGxeRq5oOej5DTg4Gl2RTTpx/QTBYcHsu7KP9r7t+aDxB6zbn0SXb7fg6uTA5N4P8UTD8vK0p5BAFw8oMx0WPgvJ1+C5P8DNy+iKbE6GKYPvD3zPl7u/xN3ZnY9bfswjfo+glKJ88Su0rmZuplWqqDyFK8wk0MWD+WMMnNkCj8+CMnWMrsbmHL92nOCwYPbH7adDxQ6MCHyPeWFX2X/0KMMfkWZa4vYk0MX927cIwr+AoBeg7pNGV2NTso/KizgXYUqrKZRUDzPg672ciE2kT6A00xJ3JoEu7s+lA7D8VfBtAh0/NLoam3Ls2jGCw4I5EHeAjhU78kb9kXy7MZYftm2jnGdhfhgcRKsAWRxE3FmOAl0p1Qn4DHAEZmutJ91mnz7AWEADe7TW/SxYpygIkq/DL/3BtSg88b25k6LItQxTBt/t/46Ze2bi4eLBlFZTeMTvEY5dusHPkWd5unFFRnSqThFXGX+Ju7vnT4hSyhH4AugAxADblVLLtdYHs+3jD7wHNNNaX1NKyfwpW2MywZKhEB8Dz/wGHmWMrsgmHL12lOCwYA7GHeQRv0d45aF32Ho0BfzMzbQ2v9OG0nLTU+RQTv7KDwKOa61PAiilfgG6Awez7fM88IXW+hqA1vqypQsVBts0GY6tNS9U4dvI6GqsXropnTn75vDV3q8o6lKUqa2mYrr5EE98uZeriWk0quxFFe8iEubivuTkSVEfIDrb65is97ILAAKUUmFKqfCsSzT/oZQaqpSKUkpFxcbGPljFIv8dWWNeF7RuP3j4OaOrsXpHrh6h/2/9mbF7Bh18OzC73XyWhpVg2E878C7iyrKXm1HFW3rhiPtnqYtyToA/0BooD2xSStXRWl/PvpPWehYwCyAwMFBb6NgiL8WdMF9qKfMQdJ0mHRRzId2Uzrf7vuXrvV9T1KUon7T+hDYV2pmbacWnMOKRagxtWVmaaYkHlpNAPwdkX3KmfNZ72cUAEVrrdOCUUuoo5oDfbpEqhTHSEs3LyDk4mDsoOksv7Qd15OoRgsOCOXT1EJ0rdWZw9TcJKFnG3EyrWy0qFHeTFrci13IyFNgO+CulKimlXIC+wPJb9lmKeXSOUqok5kswJy1Yp8hvWsPy1+DyIej1LRSvaHRFVindlM7MPTPpu7Ivl5MuM63VJ9R0fJGeM/bw09/NtKqVkjAXFnHPEbrWOkMp9QqwFvO0xTla6wNKqRAgSmu9PGtbR6XUQSATGKG1jsvLwkUeC58J+xdBu9FQtZ3R1Vilw1cPExwWzOGrh+lSqQtPVn6NCSvOEHXmIC0DvGlbXSaDCctSWhtzKTswMFBHRUUZcmxxD6e3wA/doFpn86UWuW5+X9Iz0/lm3zd8s/cbPF09CW4STOxFf0YvP0BhZ0dGd61JzwY+8rSneCBKqR1a68DbbZMnFcS/xZ+Dhc+AV2XoMVPC/D4dvnqYUVtGceTaEbpW7sq7Qe/i6erJ1vQrtK9RinHdauPt4Wp0mcJGSaCL/5eRam6Hm55sfnioUFGjK7Ia6ZnpzNo3i9l7Z1OsUDGmtPyUPUd8+Hr9Bd7p5EnTKiVpWkWaaYm8JYEu/t/qkXAuCvr8CN7VjK7GahyKO8SosFEcvXaUxyo/RscyQwlZcpqTsSfo+3AFaaYl8o0EujDbORd2fAfN34Sa3YyuxiqkZ6bz9d6vmb1vNl6FvJjc4lPC95Xm2VUH8ClWmLmDg2gpzbREPpJAF3BuB/w2HCq3hrbBRldjFQ7EHSA4LJhj147RrUo33nn4HWLjHXhj+xYGNfFjxCPVcJdmWiKfyU+cvUu8AvOfhiKloNccWUbuHtIy0/hqz1fM2T+HEoVKMKnpJ1yN88fT1RPPUrD5nTaygpAwjAS6PcvMgEWDITEWhqwF9xJGV1SgHbhygFFhozh+/Tjdq3SnnvvTjJl/hutJB2hapQRVvItImAtDSaDbs79C4NRG6P4llKtvdDUF1q2j8glNPmVlhCfDDxyjjo8ncwc3kmZaokCQQLdXB36FsM8gcAjU7290NQVW9lF5j6o9eKvhcHp8vpOL8bG817k6Q5pXwkmaaYkCQgLdHl0+DEtfhvIPQ6f/LD4lMI/KZ+6ZyXf7v6NE4RJ82PhTHvNvi4ODIqR7bSoUL0xlGZWLAkYC3d6kxMP8/uDiDn3myjJyt7Evdh/BYcGciD9BjyqP42Pqw7s/xRDf5QxPN/GTdT1FgSWBbk9MJvj1Rbh2GgatgKLljK6oQEnNTOXL3V/y/YHv8S7sTXDgNH7Z6MaPZ8/Qupo37WqUNrpEIe5KAt2ebJkKR34zX2ap2NToagqUvbF7CQ4L5mT8SXr598KXJxn182ncXRP55Mm69KgnzbREwSeBbi+O/Ql/jYc6T0CjYUZXU2CkZqbyxe4v+OHAD5RyK8VX7b+imU8ztp64QsdayYztVouSRaSZlrAOEuj24NppWDwESteCxz6TDopZ9sTuITgsmFPxp+hRpSdO8d3YvNedZj5IMy1hlWS+la1LSzIvI4eGJ3803wy1cykZKUyLmsbTq58mOSOZN2t/zJbwNny3+SI3UtIxao0AIXJLRui2TGtY+SZc3A/9F5p7nNu53Zd3ExwWzOmE03Sv3JP0K10IWXgFXy/Nz881omlVGZUL6yWBbssiv4G9v0CbD8C/g9HVGColI4UZu2Yw9+BcyriXYVaHWXg71aHr75t5rnkl3uoYgJuL/HEQ1k1+gm3VmW2w9j0I6AwthhtdjaGyj8q7Ve5FJYc+NClXE4DN77SVFYSEzZBAt0UJF2DhIChWEXp+DQ72easkOSOZGbtm8OPBHynrXpbBVSbx03oXElJO0zbAl8reRSTMhU2RQLc1GWnmME+9AQOXQiFPoysyxK7LuwgOC+ZMwhkeq9SLC6fa8dnKBB4qX5h5vRvJY/vCJkmg25q170N0BPSeA6VrGl1NvkvOSObzXZ/z08GfKFekHLPaf8O7PydzMf4mH3SpwbPN/KSZlrBZEui2ZPf/YPs30OQVqN3L6Gry3c5LOwkOC+bsjbN09evF+41H4OHqTmj3WHy93PArKVM2hW2TQLcVF/bAyjfArwW0H2d0NfkqOSOZ6TunM+/QPMoVKUfPsqHM/9OV6s6xDGrqLut6CrshgW4Lkq6aHx5yKwG9vwNH+/m27ri0g+CwYKJvRNPZtzeHDjbnh+0ptKteko61pJmWsC/28yffVpkyzY/137gIz66BIvYxGk1KT2L6run8fOhnfIr48FSFCXz3pyMehUx81rce3eqWk2Zawu5IoFu79ePhxF/w2HQo39DoavJF1MUoRm8dTfSNaPpV78frDV5nb3Qyl+qcZXTXmpSQZlrCTkmgW7NDK2DzVGgwCBoOMrqaPJeUnsRnOz/j58M/41OkPG09x8DVWrg5u9G4shuNK8si18K+yfwtaxV71LxYRbkG0OVjo6vJc9svbqfX8l78fPhn2pbtRdLJ11kWXpik1ExppiVEFhmhW6PUG+Zl5JxczR0UnWz3EkNSehKf7PiEX478gk+R8jQpHMyyv9ypWMKVn5+vIy1uhchGAt3aaA1LX4K44/D0MvAsb3RFeSbyQiSjt47m/M3zDKgxgK4VhtD7yyiGtqzIm+0DKOziaHSJQhQoOQp0pVQn4DPAEaLvwdoAABY4SURBVJittb7tUvFKqV7AIuBhrXWUxaoU/y/sMzi0HDp+CJVaGl1NnkhKT2LajmnMPzIfH/cK9PGZyMigRwHYMrKN3PQU4g7ueQ1dKeUIfAF0BmoCTyml/vNMuVLKA3gdiLB0kSLLifWwbhzUetz8NKgNirgQQc/lPVlwZAFNSz7OxUMv8cN6xcnYmwAS5kLcRU5uigYBx7XWJ7XWacAvQPfb7BcKfASkWLA+8bfrZ2HRYChZDbrNsLll5BLTE/kw/EOe+/050A5U0++ydnMj/LyK89trLaSZlhA5kJNLLj5AdLbXMUCj7DsopRoAFbTWvymlRtzpg5RSQ4GhAL6+vvdfrb1KTzY/CWrKgL7zwNW2wi38QjhjwsZwIfECA2oMZOWGupy5oQnuWo1nmvrh6GBbf3kJkVdyfVNUKeUATAOeude+WutZwCyAwMBAmWuWE1rDb2+be7U89QuUqGJ0RRaTmJ7I1KipLDy6EB93X7575AcalqlPs+JX8PVyw7eEm9ElCmFVchLo54AK2V6Xz3rvbx5AbWBD1qPWZYDlSqlucmPUAqLmwO550PIdqNbZ6GosZtv5bYzZOoaLiRep79mDiB0Ps69sMRqWgeb+MhVRiAeRk0DfDvgrpSphDvK+QL+/N2qt44F//gQqpTYAwyXMLSA6ElaPhKodoPW7RldjETfTbjJ1x1QWHV1EOTdfyiQOZ9OhEnSoWZrOdcoaXZ4QVu2ega61zlBKvQKsxTxtcY7W+oBSKgSI0lovz+si7dKNS7DgafD0gZ6zwMH651xvPb+VsVvHcinpEkHFe7IxogGehdyY0a8Wj9YpK820hMilHF1D11qvAlbd8t7oO+zbOvdl2bnMdFj0LCRfh+f+ADcvoyvKlZtpN5kSNYXFxxZTybMSczvPJeVGeYomRxPctSZe7i5GlyiETZAnRQuiP0bDmTDo+Q2UqWN0NbkSdi6MsdvGcjnpMtUKdaOucx/qetcFb2gkzbSEsCgJ9IJm70II/xIavQgP9TG6mgd2I+0GU6KmsOTYEsoUrojbldeJulya2k2d0FrL5RUh8oAEekFycT8sfxV8m0LHUKOreWBbzm1h7NaxxCbHUtn5MfbsCqJSiWIseOEhgipZ9+UjIQoyCfSCIvmauYNi4WLwxPfg6Gx0RfctIS2BKdun8OvxX6niWYW3643n7R+vMaylH2+096eQs/Xf2BWiIJNALwhMJlj8PMSfg2dXgYf1rYW5OWYzY7eN5UrSFR4u1puZXd/F1dGVRiPT5KanEPlEAr0g2DgJjv8Bj06FCkFGV3NfEtIS+Hj7xyw9vpRSrhXhwqtsOerD+WYZVCrpKmEuRD6SQDfakdWw8SOo1x8ChxhdzX3ZFLOJcVvHcSUljjL6UY7taUKDCiWZPPghKpV0N7o8IeyOBLqR4k7AkhegbF3z6NxKZn7Ep8Yzeftklp9YTtViVUmKGci562UY27UaA5tIMy0hjCKBbpTUm+YOig6O8ORP4FzY6IpyJPuo/Pk6QxlW9wW2n0rA18uNCl7STEsII0mgG0Fr8/TE2MMwYDEUK/ithLOPyr2cK5Jy5mU8qrTHxdGFZlWlmZYQBYEEuhG2fQEHlkD7sVClrdHV3NOG6A2EbAshLuUqnqmdOXOoGY/U8uFRaaYlRIEigZ7fTm0yP9pfoxs0e8Poau4qPjWejyI/YsXJFXi7+JF06imcHSsxs38t6YwoRAEkgZ6f4mNg4bPmRSp6fFmgb4KuP7uekPAQrqdcZ1jdYdQr0osljpcI7lqDYm4yFVGIgkgCPb+kp8D8gZCRCk/OA1cPoyu6rfjUeCZGTuS3k7/h6ViRdp7DebneowA0q1rG4OqEEHcjgZ5fVr8D53eaZ7R4BxhdzW39dfYvQraFcC31Oq43OnHuXHOKNPGTZlpCWAkJ9Pyw4wfY+QM0fwtqPGZ0Nf9xPeU6EyMnsurUKjyULzdO9MevqD8zX3iIh/2kmZYQ1kICPa/F7IBVw6FyG2g7yuhq/mPd2XWEbgslPjWep/yfY96aqgxrWoXX2kkzLSGsjQR6XroZCwsGQpEy0HtOgVpG7lrKNSZGTmT1qdV4u1Tml65fU82rGi/WTaO49F8RwipJoOeVzAzzMnJJcTB4bYFaRu7PM38SGh7K9dQE1LVOxFxphUuH8gAS5kJYMQn0vLJuLJzeDD1mQrl6RlcDmEflEyImsOb0GgprX26cGkiDMjWZ9Jo00xLCFkig54X9S2Dr5/Dwc1Cvn9HVAPDHmT/4MPxDEtIScL3xKImXWzCucy36N6qIgzTTEsImSKBb2qWDsOwVKB8Ej0w0uhquplxlQsQE1p5eSw2vGnzT8RuuXPXC18uN8sWlmZYQtkQC3ZKSr5s7KLq4Q5+54GTs9ejfT//Oh+EfEp96g4wrnehYZSgBxf0JKG5oWUKIPCKBbikmE/w6DK6fgUEroKhxvU7ikuOYEDGB38/8jnOGLzfODqJzQH261y34XR2FEA9OAt1SNk+Fo6uh82So2NSwMtacXsOE8AkkpN4gLbYTrqnt+fKJunSqLY/tC2HrJNAt4dgfsH48PPQkBA01pIS45DjGR4znjzN/ULtEbd6sPZyth5wY9WhNPN2cDalJCJG/JNBz6+pJWDwESteGrp/mewdFrTVrT6/lw/Dx3Ei7yUNu/fihywicHJx4vHa+liKEMJgEem6kJZk7KKLgyR/BJX9njVxJvsL48PH8efZPHNIqciPmWao/3BhHVXCeSBVC5B8J9AelNax4DS4dgP6LwKtSPh5as+b0GsaHj+dGWhKplzpTwakTcwbXp2FFmcIihL2SQH9QEV/DvoXQZhT4t8+3w15JvsKH4R+y7uw6AorV4vqJLgwLasTLbavi6iQjcyHsWY4CXSnVCfgMcARma60n3bL9LeA5IAOIBQZrrc9YuNaC43QY/P4BVOsCLd7Ol0NqrVl1ahXjwyeSlJHEmw3eZFCtQdzsYJKbnkIIIAeBrpRyBL4AOgAxwHal1HKt9cFsu+0CArXWSUqpF4HJwJN5UbDhEs7DwmegWEV4/CtwcMjzQ15JvkLotlD+iv4LUiqSdvF52nTpjaODI55uMioXQpjlJI2CgONa65Na6zTgF6B79h201uu11klZL8OB8pYts4DISIMFT0NaIvSdB4U88/RwWmtWnlzJY792Z/3ZzaRc6kJ13mP1S72kmZYQ4j9ycsnFB4jO9joGaHSX/YcAq2+3QSk1FBgK4OtrhU8trn0PYrbDE99DqRp5eqjYpFhCwkPYEL0BxzQ/TJf6MKZ9S/oF+UozLSHEbVn0pqhSagAQCLS63Xat9SxgFkBgYKC25LHz3K55sH02NH0Naj2eZ4f5e1Q+IWIS6aZUhgcOp6prZyqV9KBcscJ5dlwhhPXLSaCfAypke10+671/UUq1Bz4AWmmtUy1TXgFxfjesfBMqtYR2Y/LsMJeTLjNuawibzm3ElFyRwQHvMqhW8zw7nhDCtuQk0LcD/kqpSpiDvC/wrybfSqn6wNdAJ631ZYtXaaTEOPPDQ+7e0Ps7cLT8TE+tNStOrjDPYElPJeXyozxS/gkGBcmjnkKInLtnOmmtM5RSrwBrMU9bnKO1PqCUCgGitNbLgY+BIsBCZX70/azWulse1p0/TJmweDDcvAiD14B7SYsf4nLSZUK2hbAxZiOZSRVxT+jPtMfa0qFmaYsfSwhh23I03NRarwJW3fLe6Gxf59+TNfnpr1A4uQG6fQ4+DS360Vprlp9YzkfbPyI9M52nqrxKwqUg3numFp6FZV65EOL+yZOid3JwGWz5BBo+Aw2etuhHX0q8RHDYWLZd2EJJp+r8r9sUKhataNFjCCHsjwT67cQegaUvmUflnSdb7GO11iw7sYwJ4ZNITk8jNfYx2tfqj6+HFU7hFEIUOBLot0pJMC8j51QI+vwITq4W+diLiRcZtWUsERfDyEisRNn0p5natx31faWZlhDCMiTQs9Malr4IcSfg6WXg6WOBj9QsPb6Uydsnk27KgCs9eL5Of15pG4CLU963DRBC2A8J9Oy2fAKHV8IjE6BSi1x/3MXEi7y/aTTbL28jsHQgIU1DKOpcRm56CiHyhAT6346vM89qqdUTGr+Uq4/SWrPk2BImRkwmNSOdzLgeBHd5hwpFPSxUrBBC/JcEOsC1M+Zl5LyrQ/cZuVpG7sLNC4zcFMyu2AgyEitTw2kI055ph5800xJC5DEJ9PRk801Qkwme/AlcHix4/x6Vfxz1MYmp6eirjzOq2WCeCqoozbSEEPnCvgNda3OPlot74an5UKLKA33MhZsXGLFxFHuuRBJUJogeFd4k0KcKZT2lmZYQIv/Yd6Bvnw17/get3oVqne77l2utmX94IZO3TyEtM5N2pYbxSccXcVAye0UIkf/sN9DPRsCa98C/I7Qaed+//PzN87z11wccuBZFRmIVWhR/iTGtW0uYCyEMY5+BfuOSeeUhz/LQc9Z9LSOntWbh0YVMiphCWkYmrglP8EnHobSvWSYPCxZCiHuzv0DPTIeFgyA1AQYshsI5f1Lz3M1zjAkbQ8TFCGoWa0jZjKcJ6deCooVkXrkQwnj2F+i/j4Kz26DXt1AmZ/3GTdrEjwfm88mOaShgdJPR9PbvjcrF9EYhhLA0+wr0PfMh4ivzg0N1eufol8TciOG1P97n2I1dZCb609P3DXr7t5AwF0IUOPYT6Bf2worXoWIz6BByz91N2sR3e3/m892fkpEJxVP68XnXYdSTZlpCiALKPgI96ar54aHCxeCJ78Hx7te8Y27EMGbrGCIvRkJSAE9VepsR7RtLMy0hRIFm+4FuyoTFz0HCeXh2NRQpdeddtYlZu37i6/2fU8jJiXFNx9GufFc8C7vkY8FCCPFgbD/QN0yEE+ug6ydQ4eE77nY2Pppha98hOnk/psQAZj32MQ9XqJyPhQohRO7YdqAfXgWbPob6A6Dhs7fdxaRNfLFjLt/s/xyTSVFBD+KrPi9SUZppCSGsjO0G+pXj8OsLUK4+dJl62w6K0QnRjAoLZuflHZBcjTfqfsCQJvVkBosQwirZZqCn3oT5/cHBCfrMBedC/9ps0iamb/+OeUe/wtnBmWcDRtKvZi/KSDMtIYQVs71A1xqWvQxXjsKAJVDs3wswH792imFrRnIp7RB+boF802UiZdzlsX0hhPWzvUDf+jkcXArtx0GVNv+8bdImJm+dzbxjX6FNjtQuNJQvOw/Fy90yi0ALIYTRbCvQT26EP8dAjW7Q7PV/3j6TcIahq97hfOpBHFNrMqbpaB6vU8vAQoUQwvJsJ9CvR8OiZ6GEP/T4EpQi05TJvEPz+HzX5zgoJ+oXHsaMPs9TVOaVCyFskG0EenoKLBgIGWnQdx64erDv8nFeXjuSa6ajtCrfitFNRlPK7c4PFQkhhLWzjUBfPQLO74In55HpVZngdTNYET0HbXKiefFXmN7meRzuo+e5EEJYI+sP9B3fw8650GI4O4v58+qPvUngOIUy6jCtbSgtKj/YOqFCCGFtrDvQY6Jg1QgyK7flp9Llmf5nf9IyHWhb6g0+7vQMLk6ORlcohBD5xnoD/eZlmD+QYx6leclJc3HHNFpXaM3bDd7Hr1hZo6sTQoh8l6MLy0qpTkqpI0qp40qpd2+z3VUpNT9re4RSys/Shc7ZP4fIC5HmF5kZZCwcxDjXNHp6OnAh5QJv1xvH9DbTJcyFEHbrnoGulHIEvgA6AzWBp5RSNW/ZbQhwTWtdFfgE+MjShdYuUZvhG4cTuXcuh2a3p1fGSRZ5uOFOFeZ1WsQzdXtKDxYhhF3LySWXIOC41vokgFLqF6A7cDDbPt2BsVlfLwJmKKWU1lpbqtCgskFMqfkcr+74iBRXhcaZDu6tmdLzM5nBIoQQ5OySiw8Qne11TNZ7t91Ha50BxAMlbv0gpdRQpVSUUioqNjb2vosNio+lQ2IyJqUYkHCTaaX9JMyFECJLvqah1nqW1jpQax3o7e19378+0tObTW6FeeF6AiuLuBPpef+fIYQQtionl1zOARWyvS6f9d7t9olRSjkBnkCcRSrMEnkhkuEHZzMlcCRB8bEEeXqbX3tXJ6hskCUPJYQQVikngb4d8FdKVcIc3H2BfrfssxwYBGwDegN/WfL6OcD+uP1MaTXln/AOAqZ4V2d/3H4JdCGEAFROclcp1QX4FHAE5mitxyulQoAorfVypVQh4EegPnAV6Pv3TdQ7CQwM1FFRUbk+ASGEsCdKqR1a68DbbcvRg0Va61XAqlveG53t6xTgidwUKYQQIndkiogQQtgICXQhhLAREuhCCGEjJNCFEMJG5GiWS54cWKlY4MwD/vKSwBULlmMN5Jztg5yzfcjNOVfUWt/2qUrDAj03lFJRd5q2Y6vknO2DnLN9yKtzlksuQghhIyTQhRDCRlhroM8yugADyDnbBzln+5An52yV19CFEEL8l7WO0IUQQtxCAl0IIWxEgQ70grA4dX7LwTm/pZQ6qJTaq5Rap5SqaESdlnSvc862Xy+llFZKWf0Ut5ycs1KqT9b3+oBS6uf8rtHScvCz7auUWq+U2pX1893FiDotRSk1Ryl1WSm1/w7blVJqetbvx16lVINcH1RrXSD/w9yq9wRQGXAB9gA1b9nnJeCrrK/7AvONrjsfzrkN4Jb19Yv2cM5Z+3kAm4BwINDouvPh++wP7AKKZ70uZXTd+XDOs4AXs76uCZw2uu5cnnNLoAGw/w7buwCrAQU0BiJye8yCPEL/Z3FqrXUa8Pfi1Nl1B37I+noR0E4ppfKxRku75zlrrddrrZOyXoZjXkHKmuXk+wwQCnwEpORncXkkJ+f8PPCF1voagNb6cj7XaGk5OWcNFM362hM4n4/1WZzWehPm9SHupDswV5uFA8WUUmVzc8yCHOgWW5zaiuTknLMbgvlveGt2z3PO+qdoBa31b/lZWB7Kyfc5AAhQSoUppcKVUp3yrbq8kZNzHgsMUErFYF5/4dX8Kc0w9/vn/Z5ytMCFKHiUUgOAQKCV0bXkJaWUAzANeMbgUvKbE+bLLq0x/ytsk1Kqjtb6uqFV5a2ngO+11lOVUk2AH5VStbXWJqMLsxYFeYR+P4tTk1eLU+eznJwzSqn2wAdAN611aj7Vllfudc4eQG1gg1LqNOZrjcut/MZoTr7PMcByrXW61voUcBRzwFurnJzzEGABgNZ6G1AIcxMrW5WjP+/3oyAH+j+LUyulXDDf9Fx+yz5/L04NebQ4dT675zkrpeoDX2MOc2u/rgr3OGetdbzWuqTW2k9r7Yf5vkE3rbU1L0ibk5/tpZhH5yilSmK+BHPXdXoLuJyc81mgHYBSqgbmQI/N1yrz13Lg6azZLo2BeK31hVx9otF3gu9xl7gL5pHJCeCDrPdCMP+BBvM3fCFwHIgEKhtdcz6c85/AJWB31n/Lja45r8/5ln03YOWzXHL4fVaYLzUdBPZhXnjd8Lrz+JxrAmGYZ8DsBjoaXXMuz/d/wAUgHfO/uIYAw4Bh2b7HX2T9fuyzxM+1PPovhBA2oiBfchFCCHEfJNCFEMJGSKALIYSNkEAXQggbIYEuhBA2QgJdCCFshAS6EELYiP8DeO6iCQ8rps8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "rf_fpr, rf_tpr, thresholds = roc_curve(y_test, y_predicted)\n",
    "\n",
    "pyplot.plot([0, 1], [0, 1], linestyle='--')\n",
    "pyplot.plot(rf_fpr, rf_tpr, marker='.')\n",
    "pyplot.plot(blr_fpr, blr_tpr, marker='x')\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7325996677740865"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(y_test, y_predicted)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}