{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "**DeapSECURE module 4: Deap Learning**\n", "\n", "# Session 2: Deep Learning\n", "\n", "Welcome to the DeapSECURE online training program!\n", "This is a Jupyter notebook for the hands-on learning activities of the\n", "[\"Deap Learning\" module](https://deapsecure.gitlab.io/deapsecure-lesson04-nn/),\n", "Please visit the [DeapSECURE](https://deapsecure.gitlab.io/) website to learn more about our training program.\n", "\n", "In this session, We will use this notebook to prepare the Sherlock dataset for the DL lesson\n", "\n", "## Data Preparation\n", "\n", "When preparing data for analytics and machine learning, up to two-thirds of the time is actually spent preparing the data.\n", "This may sound like a waste of time, but that step is absolutely crucial to obtaining trustworthy insight from the data.\n", "The goal of **data preparation** is to achieve a clean, consistent and processable state of data.\n", "\n", "In this session, you will perform data preparation used in the previous ML workshop.\n", "\n", "**QUICK LINKS**\n", "* [Setup](#sec-setup)\n", "* [Loading Sherlock Data](#sec-load_data)\n", "* [Traditional Machine Learning](#sec-ML)\n", "* [Deep Neural Network](#sec-NN)\n", "* [Parallel Computing](#sec-Par)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## 1. Setup Instructions\n", "\n", "If you are opening this notebook from the Wahab OnDemand interface, you're all set.\n", "\n", "If you see this notebook elsewhere, and want to perform the exercises on Wahab cluster, please follow the steps outlined in our setup procedure.\n", "\n", "1. Make sure you have activated your HPC service.\n", "2. Point your web browser to https://ondemand.wahab.hpc.odu.edu/ and sign in with your MIDAS ID and password.\n", "3. Create a new Jupyter session using \"legacy\" Python suite, then create a new \"Python3\" notebook. (See ODU HPC wiki for more detailed help.)\n", "4. Get the necessary files using commands below within Jupyter:\n", "\n", " mkdir -p ~/CItraining/module-nn\n", " cp -pr /shared/DeapSECURE/module-nn/. ~/CItraining/module-nn\n", " cd ~/CItraining/module-nn\n", "\n", "The file name of this notebook is `NN-session-2.ipynb`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.1 Reminder\n", "\n", "* Throughout this notebook, `#TODO` is used as a placeholder where you need to fill in with something appropriate. \n", "\n", "* To run a code in a cell, press `Shift+Enter`.\n", "\n", "* Pandas cheatsheet\n", "\n", "* Summary table of the commonly used indexing syntax from our own lesson.\n", "\n", "* Keras API document\n", "\n", "We recommend you open these on separate tabs or print them;\n", "they are handy help for writing your own codes." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.2 Loading Python Libraries\n", "\n", "Next step, we need to import the required libraries into this Jupyter Notebook:\n", "`pandas`, `numpy`,`matplotlib.pyplot`,`sklearn` and `tensorflow`.\n", "\n", "**For Wahab cluster only**: before importing these libraries, we have to load the `DeapSECURE` environment module:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\n", "The following have been reloaded with a version change:\n", " 1) libjpeg-turbo/2.0.2 => libjpeg-turbo/2.0.3\n", "\n", "\n" ] } ], "source": [ "# Run to load environment modules on HPC\n", "module(\"load\", \"DeapSECURE\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Few additional modules need to be loaded to access the GPU via CUDA and TensorFlow library.\n", "Keras is now part of TensorFlow:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\n", "\n", "The following have been reloaded with a version change:\n", " 1) py-numpy/1.17.3 => py-numpy/1.16.3\n", "\n", "\n", "\n", "Currently Loaded Modules:\n", " 1) intel-mkl/2019.4.243 22) py-cycler/0.10.0\n", " 2) texlive/2020 23) py-kiwisolver/1.1.0\n", " 3) zlib/1.2.11 24) py-pyparsing/2.4.2\n", " 4) libpng/1.6.37 25) libjpeg-turbo/2.0.3\n", " 5) bzip2/1.0.8 26) py-pillow/6.2.0\n", " 6) freetype/2.10.1 27) py-matplotlib/3.1.1\n", " 7) xz/5.2.4 28) py-scipy/1.3.1\n", " 8) libtiff/4.0.10 29) py-seaborn/0.11.1\n", " 9) openjpeg/2.3.1 30) py-pip/19.3\n", " 10) python/3.7.3 31) py-joblib/0.14.0\n", " 11) py-markupsafe/1.0 32) py-scikit-learn/0.22.2.post1\n", " 12) py-babel/2.6.0 33) gmp/6.1.2\n", " 13) py-jinja2/2.10 34) mpfr/4.0.2\n", " 14) py-six/1.12.0 35) mpc/1.1.0\n", " 15) py-jupyter/1.1.4 36) DeapSECURE/2020\n", " 16) py-python-dateutil/2.8.0 37) cuda/10.2.89\n", " 17) py-pytz/2019.3 38) hdf5/1.10.5\n", " 18) py-numexpr/2.7.0 39) py-numpy/1.16.3\n", " 19) py-bottleneck/1.2.1 40) py-h5py/2.9.0\n", " 20) py-pandas/0.25.1 41) py-tensorflow/1.13.1\n", " 21) py-setuptools/41.4.0\n", "\n", " \n", "\n", "\n" ] } ], "source": [ "module(\"load\", \"cuda\")\n", "module(\"load\", \"py-tensorflow\")\n", "module(\"list\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can import all the required modules into Python:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "import os\n", "import pandas as pd\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import sklearn\n", "from sklearn import preprocessing\n", "\n", "import tensorflow as tf\n", "import tensorflow.keras as keras\n", "\n", "%matplotlib inline" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "# tools for machine learning:\n", "from sklearn import preprocessing\n", "from sklearn.model_selection import train_test_split\n", "# for evaluating model performance\n", "from sklearn.metrics import accuracy_score, confusion_matrix\n", "# classic machine learning models:\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.tree import DecisionTreeClassifier" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "# Import KERAS objects\n", "from tensorflow.keras.models import Sequential\n", "from tensorflow.keras.layers import Dense\n", "from tensorflow.keras import optimizers" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## 2. Loading Sherlock Application Dataset\n", "\n", "First of all, let us review the data preparation step, data wrangling step and machine learning step one by one in this bigger dataset. In the first step above we are actually using two Python scripts: `Prep_ML.py` and `analysis_sherlock_ML.py`\n", "\n", "The script `Prep_ML.py` contains all the steps necessary to read the data, remove useless data, handle missing data, extract the feature matrix and labels, then do the train/dev split. Load the commands contained in this script into your current Jupyter notebook using the IPython’s `%load` magic. Then you can run this function.\n", "\n", "The script `analysis_sherlock_ML.py` is a library of functions, which contains the steps we described in the earlier lesson. These functions are clearly named such as: `preprocess_sherlock_19F17C`, `step0_label_features`, `step_onehot_encoding`, and `step_feature_scaling`. \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Uncomment and run the magic statement `%load Prep_ML.py` below.\n", "(It will replace the cell with the contents of Prep_ML.py.)\n", "You may have to run this cell twice with `Shift+Enter` to actually run the loaded code." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "#%load Prep_ML.py\n", "\"\"\"^^^ Uncomment and run the magic statement above.\n", " You may have to run the cell twice to actually run this cell!\"\"\";" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "After the cell above is executed, you will find the training & test data in the following members of `Rec` object:\n", "\n", "* `Rec.df_features`: DataFrame of the features for the machine learning models\n", "* `Rec.labels`: The labels (expected output of the ML models)\n", "* `Rec.train_features` = training data's features\n", "* `Rec.test_features` = testing data's features\n", "* `Rec.train_labels` = training data's labels\n", "* `Rec.test_labels` = testing data's labels\n", "\n", "We use this approach to manage the complexity of having too many variables (e.g. `train_F`, `train_F2`, `train_F3`, ...)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.1 About the SherLock \"18-apps\" Dataset\n", "\n", "This is a more diverse subset of the SherLock Application dataset, covering significantly more applications and features.\n", "\n", "> Your challenge is to train a similar model (like in the previous notebooks) using the \"18-apps\" dataset to correctly classify running apps on the smartphone with very high accuracy (> 99%)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**EXERCISE**\n", "\n", "Take a peek at the training feature DataFrame." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "\"\"\"Take a peek at the training feature DataFrame.\"\"\";\n", "#TODO" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "From above, we know that we are working with a significantly larger data file, `sherlock/sherlock_18apps.csv`.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question:**\n", "\n", "- Please check out the `analysis_sherlock_ML.py` and see how this function defined.\n", "- How many features for each record?\n", "- How many applications in the total dataset?\n", "- How many records in the seperated training and testing dataset?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This dataset has 19 features for each record and 18 applications in total." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## 3. Traditional Machine Learning \n", "\n", "Now, we first try the traditional machine learning algorithms we learn in the previous session. \n", "Here we test on **Decision Tree** and **Logistic Regression**. \n", "To simplify the code, we will use the `model_evaluate` function to evaluate the performance of a machine learning model (whether traditional ML or neural network model)." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "def model_evaluate(model,test_F,test_L):\n", " test_L_pred = model.predict(test_F)\n", " print(\"Evaluation by using model:\",type(model).__name__)\n", " print(\"accuracy_score:\",accuracy_score(test_L, test_L_pred))\n", " print(\"confusion_matrix:\",\"\\n\",confusion_matrix(test_L, test_L_pred))\n", " return" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 1.29 s, sys: 16.4 ms, total: 1.3 s\n", "Wall time: 1.31 s\n" ] }, { "data": { "text/plain": [ "DecisionTreeClassifier(ccp_alpha=0.0, class_weight=None, criterion='entropy',\n", " max_depth=6, max_features=None, max_leaf_nodes=None,\n", " min_impurity_decrease=0.0, min_impurity_split=None,\n", " min_samples_leaf=1, min_samples_split=8,\n", " min_weight_fraction_leaf=0.0, presort='deprecated',\n", " random_state=None, splitter='best')" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ML_dtc = DecisionTreeClassifier(criterion='entropy',\n", " max_depth=6,\n", " min_samples_split=8)\n", "%time ML_dtc.fit(Rec.train_features, Rec.train_labels)" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Evaluation by using model: DecisionTreeClassifier\n", "accuracy_score: 0.9497216932766954\n", "confusion_matrix: \n", " [[ 1829 1 0 0 0 0 0 0 0 0 0 18 0 0 0 0 1 0]\n", " [ 0 5477 0 0 0 69 0 0 0 0 0 0 0 5 0 0 2 0]\n", " [ 1 610 2753 0 0 25 0 5 0 1 1 1 0 2 0 0 0 0]\n", " [ 0 0 0 4029 0 0 15 0 0 0 0 0 0 0 0 0 0 10]\n", " [ 0 0 0 0 4006 0 0 0 0 0 0 0 0 0 0 0 0 0]\n", " [ 64 28 0 0 0 3183 1 0 0 0 0 1 0 49 0 0 0 0]\n", " [ 0 143 0 0 0 2 10459 0 0 0 15 0 0 0 0 0 1369 0]\n", " [ 0 58 0 0 0 24 4 1408 0 1 0 0 0 1 0 0 11 0]\n", " [ 3 39 0 0 0 0 1 0 935 0 0 0 0 0 1 0 4 0]\n", " [ 0 0 0 0 0 0 1 0 0 486 0 0 0 8 0 0 0 0]\n", " [ 0 0 0 0 0 0 0 0 0 0 4016 0 0 0 0 0 0 0]\n", " [ 0 0 0 0 0 0 0 0 0 0 0 1697 0 0 0 0 0 0]\n", " [ 0 13 0 0 4 0 0 0 0 0 0 0 680 1 0 0 0 0]\n", " [ 0 0 0 0 0 0 0 0 0 0 0 6 0 3473 0 0 0 0]\n", " [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1003 0 0 0]\n", " [ 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 1642 0 0]\n", " [ 0 4 0 0 0 0 4 0 0 0 0 0 0 0 0 0 3897 0]\n", " [ 0 0 0 0 0 116 0 0 0 0 0 0 0 0 0 0 0 897]]\n" ] } ], "source": [ "model_evaluate(ML_dtc, Rec.test_features, Rec.test_labels)" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/shared/apps/auto/py-scikit-learn/0.22.2.post1-gcc-7.3.0-wpia/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):\n", "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n", "\n", "Increase the number of iterations (max_iter) or scale the data as shown in:\n", " https://scikit-learn.org/stable/modules/preprocessing.html\n", "Please also refer to the documentation for alternative solver options:\n", " https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n", " extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n" ] }, { "data": { "text/plain": [ "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n", " intercept_scaling=1, l1_ratio=None, max_iter=100,\n", " multi_class='auto', n_jobs=None, penalty='l2',\n", " random_state=None, solver='lbfgs', tol=0.0001, verbose=0,\n", " warm_start=False)" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ML_log = LogisticRegression(solver='lbfgs')\n", "%time ML_log.fit(Rec.train_features, Rec.train_labels)" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Evaluation by using model: LogisticRegression\n", "accuracy_score: 0.9197854108686099\n", "confusion_matrix: \n", " [[ 1387 3 63 0 0 319 0 0 0 0 0 0 0 72 5 0 0 0]\n", " [ 0 4590 390 0 0 37 77 10 6 64 0 0 0 72 31 273 3 0]\n", " [ 60 271 2817 0 0 7 13 4 0 0 0 0 0 85 141 1 0 0]\n", " [ 0 1 0 4021 0 2 11 4 0 0 5 0 0 0 0 2 0 8]\n", " [ 0 0 0 0 3999 0 0 0 0 7 0 0 0 0 0 0 0 0]\n", " [ 47 39 14 0 0 3189 24 10 1 0 0 0 0 2 0 0 0 0]\n", " [ 7 93 0 51 0 19 11628 8 0 0 29 58 0 0 0 0 93 2]\n", " [ 0 28 0 2 0 33 1 1442 0 0 0 1 0 0 0 0 0 0]\n", " [ 147 27 673 0 0 1 0 0 113 0 0 0 0 4 7 0 11 0]\n", " [ 0 0 0 0 0 0 0 0 0 433 0 0 0 24 0 38 0 0]\n", " [ 0 0 0 0 0 0 0 0 0 0 4016 0 0 0 0 0 0 0]\n", " [ 0 0 0 0 0 0 3 0 0 0 0 1642 0 52 0 0 0 0]\n", " [ 0 1 0 0 0 0 0 1 0 4 0 0 692 0 0 0 0 0]\n", " [ 17 2 239 0 0 0 0 0 17 31 0 0 0 3080 80 13 0 0]\n", " [ 99 5 172 0 0 45 0 0 7 0 0 0 0 2 673 0 0 0]\n", " [ 0 3 0 2 0 0 0 0 0 0 0 0 0 6 0 1634 0 0]\n", " [ 0 0 0 0 0 0 33 0 0 0 0 0 0 0 0 0 3872 0]\n", " [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 6 0 0 1007]]\n" ] } ], "source": [ "model_evaluate(ML_log, Rec.test_features, Rec.test_labels)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**QUESTIONS**:\n", "\n", "* Do you notice issues with the training process of any of the models above?\n", "* (Optional) Can you find a way to ensure full convergence of the training?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By now, we have a pretty good background knowledge about this dataset.\n", "And we know the accuracy scores we can get by using the Decision Tree and Logistic Regression methods,\n", "which are reasonably good, but not close to 99%." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Timing the Computation\n", "\n", "Do you notice that the training of logistic regression model takes a while?\n", "Often we want to know *how long* this actually takes place.\n", "We can get this timing easily in Jupyter by prepending `%time` to the Python statement we'd like to measure the execution time.\n", "\n", "**EXERCISE**:\n", "If you haven't already, let's retrain the logistic regression model here and get the timing:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> #### About the Warning Message\n", ">\n", "> The training phase stops with an error:\n", ">\n", "> ```\n", "> ConvergenceWarning: lbfgs failed to converge (status=1):\n", "> STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n", "> ```\n", ">\n", "> This happens because the solver fails to reach convergence after the maximum number of iteration (default=100) is reached.\n", "> You may want to investigate by trying different solvers in the `LogisticRegression` object.\n", "> Please Scikit-learn documentation on [Logistic Regression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html), the `solver` argument, if you are interested.\n", "> Our internal test showed that with another solver" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## 4. Building Neural Networks to Classify Applications\n", "\n", "Let us now proceed by building some neural network models to classify smartphone apps." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4.1 One-Hot Encoding\n", "\n", "When using neural networks to do a classification task, we need to encode the labels using **one-hot encoding**.\n", "This is necessary because many machine learning algorithms require numeric labels due to implementation efficiency, as such, any categorical data must be converted to numerical data.\n", "\n", "\n", "For more information on why we need one-hot encoding, see these articles:\n", "\n", "* https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/\n", "* https://machinelearningmastery.com/one-hot-encoding-for-categorical-data/\n", "\n", "Comment: We did not have to do one-hot in scikit-learn, because the ML objects such as `DecisionTreeClassifier` does it for us behind the scene.\n", "\n", "Similarly, any input features that are of categorical data type will also have to be encoded using either integer encoding or one-hot encoding.\n" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ "Rec.train_L_onehot = pd.get_dummies(Rec.train_labels)\n", "Rec.test_L_onehot = pd.get_dummies(Rec.test_labels)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For one-hot encoding, there is a `1` in a distinct spot for every category and `0` everywhere else.\n", "Below shows the first five rows; notice that there is only a single `1` in each row, with the rest being `0`." ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
CalendarChromeES File ExplorerFacebookGeo NewsGmailGoogle AppHangoutsMapsMessagesMessengerMoovitMoriartyPhotosSkypeWazeWhatsAppYouTube
247525000000001000000000
93942000001000000000000
22691000001000000000000
202123000000000000000001
230029000000100000000000
\n", "
" ], "text/plain": [ " Calendar Chrome ES File Explorer Facebook Geo News Gmail \\\n", "247525 0 0 0 0 0 0 \n", "93942 0 0 0 0 0 1 \n", "22691 0 0 0 0 0 1 \n", "202123 0 0 0 0 0 0 \n", "230029 0 0 0 0 0 0 \n", "\n", " Google App Hangouts Maps Messages Messenger Moovit Moriarty \\\n", "247525 0 0 1 0 0 0 0 \n", "93942 0 0 0 0 0 0 0 \n", "22691 0 0 0 0 0 0 0 \n", "202123 0 0 0 0 0 0 0 \n", "230029 1 0 0 0 0 0 0 \n", "\n", " Photos Skype Waze WhatsApp YouTube \n", "247525 0 0 0 0 0 \n", "93942 0 0 0 0 0 \n", "22691 0 0 0 0 0 \n", "202123 0 0 0 0 1 \n", "230029 0 0 0 0 0 " ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "Rec.train_L_onehot.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4.2 No Hidden Layer\n", "\n", "Here, we first give an example of a neural network model without any hidden layer." ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [], "source": [ "def NN_Model_no_hidden(learning_rate):\n", " \"\"\"Definition of deep learning model with one dense hidden layer\"\"\"\n", " model = Sequential([\n", " Dense(18, activation='softmax',input_shape=(19,),kernel_initializer='random_normal')\n", " ])\n", " adam=tf.keras.optimizers.Adam(lr=learning_rate, beta_1=0.9, beta_2=0.999, amsgrad=False)\n", " model.compile(optimizer=adam,\n", " loss='categorical_crossentropy',\n", " metrics=['accuracy'])\n", " return model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We train this model with an initial *learning rate* of 0.0003." ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /shared/apps/auto/py-tensorflow/1.13.1-gcc-7.3.0-j7tz/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Colocations handled automatically by placer.\n", "Train on 218461 samples, validate on 54616 samples\n", "WARNING:tensorflow:From /shared/apps/auto/py-tensorflow/1.13.1-gcc-7.3.0-j7tz/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.cast instead.\n", "Epoch 1/5\n", " - 10s - loss: 1.6887 - acc: 0.5517 - val_loss: 1.2622 - val_acc: 0.6810\n", "Epoch 2/5\n", " - 9s - loss: 1.1127 - acc: 0.7251 - val_loss: 1.0040 - val_acc: 0.7619\n", "Epoch 3/5\n", " - 9s - loss: 0.9270 - acc: 0.7847 - val_loss: 0.8686 - val_acc: 0.7872\n", "Epoch 4/5\n", " - 9s - loss: 0.8177 - acc: 0.8042 - val_loss: 0.7802 - val_acc: 0.8105\n", "Epoch 5/5\n", " - 9s - loss: 0.7428 - acc: 0.8178 - val_loss: 0.7168 - val_acc: 0.8216\n" ] } ], "source": [ "model_0 = NN_Model_no_hidden(0.0003)\n", "model_0_history = model_0.fit(Rec.train_features,\n", " Rec.train_L_onehot,\n", " epochs=5, batch_size=32,\n", " validation_data=(Rec.test_features, Rec.test_L_onehot),\n", " verbose=2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> To better analyze the training process, we would like to visualize model training history.\n", "> In Keras, we can collect the history with `history` function, returned from training the model and creates two charts:\n", "> - A plot of accuracy on the training and validation datasets over training epochs.\n", "> - A plot of loss on the training and validation datasets over training epochs.\n" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "dict_keys(['loss', 'acc', 'val_loss', 'val_acc'])" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_0_history.history.keys()" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [], "source": [ "def plot_loss(model_history):\n", " # summarize history for loss\n", " plt.plot(model_history.history['loss'])\n", " plt.plot(model_history.history['val_loss'])\n", " plt.title('Model Loss')\n", " plt.ylabel('loss')\n", " plt.xlabel('epoch')\n", " plt.legend(['train', 'test'], loc='upper right')\n", " plt.show()" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [], "source": [ "def plot_acc(model_history):\n", " # summarize history for accuracy\n", " plt.plot(model_history.history['acc'])\n", " plt.plot(model_history.history['val_acc'])\n", " plt.title('Model Accuracy')\n", " plt.ylabel('accuracy')\n", " plt.xlabel('epoch')\n", " plt.legend(['train', 'test'], loc='upper left')\n", " plt.show()" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3xV9f3H8dcng4RACJDBCiHMgIGgggiiMhyAuKVqrbPWWdv681dn1WptK3b91Lbu0aqtC3BvBQQVRUD23oSVsBJWQsb398e5kBgCBsi95+be9/PxuA+Te869+eRI8s73nO/5fsw5h4iIRK8YvwsQERF/KQhERKKcgkBEJMopCEREopyCQEQkyikIRESinIJA5CDMLNvMnJnF1WHfK83si1DUJVKfFAQSMcxspZntMbO0Gs/PDPwyz/anskMLFJFQUxBIpFkB/HjvJ2bWC2jsXzki4U9BIJHmReDyap9fAbxQfQczSzGzF8ys0MxWmdndZhYT2BZrZn8xs01mthwYWctrnzWz9Wa21sx+b2axR1KwmSWY2cNmti7weNjMEgLb0szsXTPbZmZbzGxytVpvD9Sw3cwWmdkpR1KHRC8FgUSar4FmZtYj8Av6IuClGvv8HUgBOgGD8ILjqsC2a4AzgWOAvsCoGq/9N1AOdAnsczrwsyOs+TdAf+BooDfQD7g7sO1/gXwgHWgF3AU4M8sBbgKOc84lA8OAlUdYh0QpBYFEor2jgtOAhcDavRuqhcOdzrntzrmVwF+BywK7XAg87Jxb45zbAjxY7bWtgBHAzc65nc65AuD/gIuPsN6fAL9zzhU45wqB+6vVUwa0ATo458qcc5Odt0BYBZAAHGVm8c65lc65ZUdYh0QpBYFEoheBS4ArqXFaCEgDGgGrqj23CmgX+LgtsKbGtr06APHA+sCpmm3Ak0DGEdbbtpZ62gY+/jOwFPjYzJab2R0AzrmlwM3AfUCBmb1iZm0ROQwKAok4zrlVeBeNzwDG1di8Ce+v7A7VnsuiatSwHmhfY9tea4BSIM051zzwaOacyz3CktfVUs+6wPey3Tn3v865TsBZwC17rwU45/7rnDsx8FoHPHSEdUiUUhBIpLoaGOqc21n9SedcBfAa8AczSzazDsAtVF1HeA34pZllmlkL4I5qr10PfAz81cyamVmMmXU2s0GHUFeCmSVWe8QALwN3m1l6YOrrvXvrMbMzzayLmRlQjHdKqMLMcsxsaOCicgmwO7BN5JApCCQiOeeWOeemHWDzL4CdwHLgC+C/wHOBbU8DHwGzgBnsP6K4HO/U0nxgKzAG7xx+Xe3A+6W99zEU+D0wDZgNzAl83d8H9u8KfBp43RTgMefcRLzrA6PxRjgb8E5P3XUIdYjsY2pMIyIS3TQiEBGJcgoCEZEopyAQEYlyCgIRkSjX4FZCTEtLc9nZ2X6XISLSoEyfPn2Tcy69tm0NLgiys7OZNu1AswJFRKQ2ZrbqQNt0akhEJMopCEREopyCQEQkyjW4awQiIoejrKyM/Px8SkpK/C4lqBITE8nMzCQ+Pr7Or1EQiEhUyM/PJzk5mezsbLw1/CKPc47NmzeTn59Px44d6/w6nRoSkahQUlJCampqxIYAgJmRmpp6yKMeBYGIRI1IDoG9Dud7jJogWLdtN/e/M4+yikq/SxERCStREwRz1hbx/JcreWrScr9LEZEotG3bNh577LFDft0ZZ5zBtm3bglBRlagJgmG5rRnZqw2PfLqEpQU7/C5HRKLMgYKgouLgjeXef/99mjdvHqyygCgKAoD7zs6lcaNYbh87m8pKNeQRkdC54447WLZsGUcffTTHHXccQ4YM4ZJLLqFXr14AnHvuufTp04fc3Fyeeuqpfa/Lzs5m06ZNrFy5kh49enDNNdeQm5vL6aefzu7du+ultqiaPpqenMC9Zx7F/74+ixe/XsUVJ2T7XZKI+OD+d+Yxf11xvb7nUW2b8duzcg+4ffTo0cydO5eZM2cyceJERo4cydy5c/dN83zuuedo2bIlu3fv5rjjjuOCCy4gNTX1e++xZMkSXn75ZZ5++mkuvPBCxo4dy6WXXnrEtUfViADg/GPbMahbOg99uJA1W3b5XY6IRKl+/fp9b67/o48+Su/evenfvz9r1qxhyZIl+72mY8eOHH300QD06dOHlStX1kstUTUiAG9q1R/O68mw/5vEXW/M4YWf9ouKKWUiUuVgf7mHSpMmTfZ9PHHiRD799FOmTJlCUlISgwcPrvVegISEhH0fx8bG1tupoagbEQBktkji9hHdmbxkE2NnrPW7HBGJAsnJyWzfvr3WbUVFRbRo0YKkpCQWLlzI119/HdLaom5EsNelx3fgnVnreODd+ZzcLY2M5ES/SxKRCJaamsrAgQPp2bMnjRs3plWrVvu2DR8+nCeeeIK8vDxycnLo379/SGsz5xrW7Jm+ffu6+mpMs6xwByMemcwp3TN4/NI+9fKeIhKeFixYQI8ePfwuIyRq+17NbLpzrm9t+0flqaG9Oqc35eZTu/LB3A18MGe93+WIiPgiqoMA4JqTOpHbthn3vDWPol1lfpcjIhJyUR8E8bEx/GlUHlt37eGB9+b7XY6ISMhFfRAA5LZN4fpBnRgzPZ9Jiwv9LkdEJKQUBAG/GNqVTulNuHPcHHaWlvtdjohIyCgIAhLjY/nTBXmsK9rNnz9a5Hc5IiIhE7QgMLPnzKzAzOYeZJ/BZjbTzOaZ2efBqqWu+ma35IoB2fx7ykqmrdzidzkiEkEOdxlqgIcffphdu4K3JE4wRwT/AoYfaKOZNQceA852zuUCPwpiLXV267Ac2qY05vaxsykpO/jysCIidRXOQRC0O4udc5PMLPsgu1wCjHPOrQ7sXxCsWg5Fk4Q4Hjy/F5c/N5V/jF/Kr4fl+F2SiESA6stQn3baaWRkZPDaa69RWlrKeeedx/3338/OnTu58MILyc/Pp6KignvuuYeNGzeybt06hgwZQlpaGhMmTKj32vxcYqIbEG9mE4Fk4BHn3Au17Whm1wLXAmRlZQW9sJO7pTOqTyaPf76MEb1ak9s2JehfU0RC6IM7YMOc+n3P1r1gxOgDbq6+DPXHH3/MmDFjmDp1Ks45zj77bCZNmkRhYSFt27blvffeA7w1iFJSUvjb3/7GhAkTSEtLq9+aA/y8WBwH9AFGAsOAe8ysW207Oueecs71dc71TU9PD0lxd4/sQYukRtw2Zjbl6nMsIvXo448/5uOPP+aYY47h2GOPZeHChSxZsoRevXrx6aefcvvttzN58mRSUkLzR6ifI4J8YJNzbiew08wmAb2BxT7WtE/zpEY8cE4uN/xnBk9PXsENgzv7XZKI1JeD/OUeCs457rzzTq677rr9tk2fPp3333+fO++8k9NPP51777036PX4OSJ4CzjJzOLMLAk4HljgYz37GdGrDcNzW/N/ny5meaH6HIvI4au+DPWwYcN47rnn2LHD+72ydu1aCgoKWLduHUlJSVx66aX8+te/ZsaMGfu9NhiCNiIws5eBwUCameUDvwXiAZxzTzjnFpjZh8BsoBJ4xjl3wKmmfvndubl89ddN3D52Nq9eO4CYGDWxEZFDV30Z6hEjRnDJJZcwYMAAAJo2bcpLL73E0qVLufXWW4mJiSE+Pp7HH38cgGuvvZYRI0bQpk2boFwsjuplqOvq9WlruHXMbB44J5fLBmSH9GuLSP3QMtRahvqIjOqTyUld0xj9wULWbquf1nAiIuFCQVAHZsYfz+uFA+4aN4eGNooSETkYBUEdtW+ZxG3Dcvh8cSFvfKc+xyINUTT8EXc436OC4BBcNiCbPh1a8Lt351O4vdTvckTkECQmJrJ58+aIDgPnHJs3byYx8dB6sEdt8/rDERtjPHRBL8545Avue2ce/7zkWL9LEpE6yszMJD8/n8LCyO45kpiYSGZm5iG9RkFwiLpkJPOrU7vy548WcXbvDQzLbe13SSJSB/Hx8XTs2NHvMsKSTg0dhmtP7kSPNs24+8256nMsIg2eguAwxMfG8OdReWzZuYc/vh9WN0OLiBwyBcFh6tkuhWtO6sSr09bwxZJNfpcjInLYFARH4OZTu9IprQl3jJvNrj3qcywiDZOC4Agkxscy+oI88rfu5i8fhcWiqSIih0xBcIT6dWzJZf078PxXK5ixeqvf5YiIHDIFQT24bXgObZolctuY2ZSWq8+xiDQsCoJ6kJwYzx/O78XSgh38c/xSv8sRETkkCoJ6MiQng/OPacdjE5exYH2x3+WIiNSZgqAe3XPmUTRPilefYxFpUBQE9ahFk0bcf3ZP5qwt4tkvVvhdjohInSgI6tkZvVpz+lGt+Nsni1mxaaff5YiI/CAFQT0zMx44tyeN4mK4Y+xsKisjd8lbEYkMCoIgaNUskbtH9uCbFVv479TVfpcjInJQCoIgubBvewZ2SWX0BwtZpz7HIhLGFARBYmY8eF4eFZWOu9+cG9FdkUSkYVMQBFFWahK/HpbD+IUFvD1rnd/liIjUSkEQZFeekM0xWc257+15bNqhPsciEn4UBEEWG2P86YI8dpZWcP878/0uR0RkPwqCEOjaKpmbhnbhnVnr+GT+Rr/LERH5HgVBiFw/qDPdWydz95tzKNqtPsciEj4UBCHSKC6GP43Ko3B7KaM/UJ9jEQkfCoIQystszjUndeLlqWv4aqn6HItIeFAQhNjNp3YjOzWJO8bNYfceNbEREf8pCEKscSOvz/HqLbv468eL/C5HRERB4If+nVL5yfFZPPflCr5Tn2MR8VnQgsDMnjOzAjOb+wP7HWdmFWY2Kli1hKM7RnSnVbNEbh87mz3lamIjIv4J5ojgX8Dwg+1gZrHAQ8BHQawjLCUnxvOH83qyeOMO/jlBfY5FxD9BCwLn3CRgyw/s9gtgLFAQrDrC2dDurTj36LY8NnEpCzeoz7GI+MO3awRm1g44D3iiDvtea2bTzGxaYWFh8IsLoXvPyiU5MZ7bx8ymQk1sRMQHfl4sfhi43Tn3g3MonXNPOef6Ouf6pqenh6C00GnZpBH3nZ3LrPwinv9SfY5FJPT8DIK+wCtmthIYBTxmZuf6WI9vzsprw6k9MvjLx4tYqT7HIhJivgWBc66jcy7bOZcNjAFudM696Vc9fjIzfn9uL+JjYrhj3Gw1sRGRkArm9NGXgSlAjpnlm9nVZna9mV0frK/ZkLVOSeSukT34evkWXvl2jd/liEgUiQvWGzvnfnwI+14ZrDoakouPa8/bM9fxx/cWMDgnnTYpjf0uSUSigO4sDiNmxugLelFWWcndb6jPsYiEhoIgzHRIbcKvT8/hs4UFvDN7vd/liEgUUBCEoasGdqR3e6/P8Zade/wuR0QinIIgDO3tc7y9pIz735nndzkiEuEUBGEqp3UyPx/ShbdmruOzBepzLCLBoyAIYzcO7kJOq2R+88Zctpeoz7GIBIeCIIw1iovhoVF5FGwvYfQHC/0uR0QilIIgzB3dvjk/HdiR/3yzminLNvtdjohEIAVBA/C/p+eQ1TKJO8fNVp9jEal3CoIGwOtz3IuVm3fx8KeL/S5HRCKMgqCBOKFzGj/u156nJy9n1pptfpcjIhFEQdCA3HlGD9KTE9TnWETqlYKgAWmWGM8fzu3Fwg3beeLzZX6XIyIRQkHQwJx6VCvO6t2Wv49fwpKN2/0uR0QigIKgAbrvrKNomhDHrepzLCL1QEHQAKU2TeC+s3OZuWYb//pqpd/liEgDpyBooM7u3Zah3TP4y0eLWL15l9/liEgDpiBooLw+xz2JjTHufEN9jkXk8CkIGrC2zRtzx4jufLl0M69NU59jETk8CoIG7pJ+WRzfsSW/f28BG4tL/C5HRBqg6AqCCDx9EhNjjL4gjz3lldz9pvoci8ihi54g2DAXnhoM62f7XUm965jWhFtO68Yn8zfy3hz1ORaRQxM9QVBSBNs3wNNDYdJfoKLc74rq1dUndqRXuxR++9Y8tqrPsYgcgugJguyBcOMU6HEmjH8Anh8BmyNnmYa42Bj+NCqPot1lPPDufL/LEZEGJHqCACCpJYx6Hs5/BjYtgidOhG+fjZhrBz3aNOPGwZ0Z991aJiwq8LscEWkgoisIAMwg70dwwxRofzy8dwv850feaaMI8POhXeia0ZTfjJujPsciUifRFwR7pbSDS8fBGX+BlV/AY/1h7ji/qzpiCXGxPDQqj/XFJfzpw0V+lyMiDUD0BgFATAz0uwaunwwtO8GYq2Dsz2D3Vr8rOyLHZrXgqhM68uLXq5i6Yovf5YhImKtTEJjZr8ysmXmeNbMZZnZ6sIsLmbSu8NOPYchvYN4b8NgJsGy831UdkV8P60Zmi8bcPnY2JWXqcywiB1bXEcFPnXPFwOlAOnAVMDpoVfkhNg4G3QZXfwIJTeHF8+D9W2FPw1zQLalRHKPPz2PFpp08/OkSv8sRkTBW1yCwwH/PAJ53zs2q9lxkaXcsXDcJ+t8IU5+CJ0+C/Ol+V3VYTuyaxkV9vT7Hc/KL/C5HRMJUXYNgupl9jBcEH5lZMnDQprlm9pyZFZjZ3ANs/4mZzQ48vjKz3odWehDFN4bhD8Llb0NZCTx7Gkz4I1Q0vFk4d43sQWqTRtw2djZlFepzLCL7q2sQXA3cARznnNsFxOOdHjqYfwHDD7J9BTDIOZcHPAA8VcdaQqfTILjhS8i7ED5/CJ45FQob1kyclMbxPHBuTxasL+ZJ9TkWkVrUNQgGAIucc9vM7FLgbuCg5xqcc5OAA05Zcc595ZzbOz3nayCzjrWEVuPmcN4TcOELsG01PHkyfP04VDacv66H5bZmZK82PPrZUpYWqM+xiHxfXYPgcWBX4PTNbcAq4IV6rONq4IN6fL/6d9Q5cOPX0GkwfHgHvHgObGs4PQDuOzuXpIRYblOfYxGpoa5BUO689Y3PAR5xzj0CJNdHAWY2BC8Ibj/IPtea2TQzm1ZYWFgfX/bwJLeCH78CZz0Ka2fA4yfArFcaxBIV6ckJ3HvmUcxYvY0Xpqz0uxwRCSN1DYLtZnYncBnwnpnF4l0nOCJmlgc8A5zjnNt8oP2cc0855/o65/qmp6cf6Zc9MmbQ5wq4/gtolQtvXAevXQY7D1h+2DjvmHYM6pbOnz5cxJotDXNarIjUv7oGwUVAKd79BBuAdsCfj+QLm1kWMA64zDm3+EjeyxctO8KV78Gp98Pij7wlKhZ96HdVB2Vm/PH8XsQY3PXGHDWxERGgjkEQ+OX/HyDFzM4ESpxzB71GYGYvA1OAHDPLN7Orzex6M7s+sMu9QCrwmJnNNLNph/9t+CQmFk68Ga6ZAE0z4OWL4O1fQGn4XpBtF+hzPHnJJsZMz/e7HBEJA1aXvwrN7EK8EcBEvBvJTgJudc6NCWp1tejbt6+bNi0MM6O81LvX4MtHoHkWnPckdBjgd1W1qqx0XPTUFBZt2M6ntwwio1mi3yWJSJCZ2XTnXN/attX11NBv8O4huMI5dznQD7invgqMCHEJcNr9cFVg8tPzI+CTe72ACDN7+xyXlFdy71vz/C5HRHxW1yCIcc5V73Sy+RBeG106DPBuQjv2cm908PRQr19ymOmc3pT/ObUbH87bwAfqcywS1er6y/xDM/vIzK40syuB94D3g1dWA5eQDGc/Cj9+FXYUwNND4IuHoTK8VgG95qSO9GzXjHvemse2XepzLBKt6nqx+Fa8JSDygN7AU865A877l4Cc4V6f5G7D4NPfwr9GwpYVfle1T1xsDA9dkMe2XXt44N0FfpcjIj6p8+kd59xY59wtzrn/cc69EcyiIkqTNLjwRe/i8cZ5Xp/k6f8Om5vQctumcP2gzoydkc/ni328WU9EfHPQIDCz7WZWXMtju5kVh6rIBs8Mel8MN3zlLXP9zi/h5Yth+0a/KwPgpqFd6JzehLvGzWFHabnf5YhIiB00CJxzyc65ZrU8kp1zzUJVZMRo3h4uewuGj4blE72b0Oa/7XdVJMbH8tAFeawr2s2fP1zodzkiEmKa+RNqMTHQ/wav+U3zLG95inHXQYm/jWP6ZrfkigHZvPD1Kr5dqT7HItFEQeCX9Bz42acw6HaY87rXJ3n5576WdOuwHNqmqM+xSLRREPgpNh6G3OX1SY5PhBfOhg/ugLLdvpTTJCGOB8/vxfLCnfx9vPoci0QLBUE4yOwD102GftfCN4/Dk4Ng3Xe+lHJyt3RG9cnkic+XM3et+hyLRAMFQbholARn/Bkue8NbtO6ZU2HiQ1AR+lk8d4/sQYukRtw2Zja79mgWkUikUxCEm85D4cavIPc8mPhHeO502BTa0zTNkxrx+3N7Mn99MSf/aQJPfr5MgSASwRQE4ahxC7jgGRj1PGxZDk+cBN88FdI+ycN7tmbsDQPo0aYZD36wkBMfmsDjE5exU/cZiEScOi1DHU7CdhnqYCle7/U4WPoJdBoC5/wTUtqFtITpq7byyGdLmLS4kBZJ8VxzcicuH5BN04S4kNYhIofvYMtQKwgaAudg+vPw0W+8mUZn/BV6jfLuWA6h71Z7gTBxUSHNk+K55qROXD6gA8mJR9y1VESCTEEQKTYvgzeuh/yp3jWEkX+DpJYhL2Pmmm08+tkSxi8sIKVxPD87sSNXDMymmQJBJGwpCCJJRTl89QhMeBCSUuGcf0DX03wpZXa+FwifLiigWWIcV5/YiatOVCCIhCMFQSRaPxvGXQuFC6DvT+G0ByChqS+lzF1bxCOfLeGT+RtJTozjpwM78tMTO5LSWIEgEi4UBJGqrATGPwBT/gktsuH8p6B9P9/Kmbu2iEc/W8LHgUC4amBHrh7YkZQkBYKI3xQEkW7lF/DGDVCcDyf+Dwy6A+Ia+VbO/HXFPPrZEj6ct4HkhDiuHJjN1Sd2pHmSfzWJRDsFQTQoKYYP74SZL0HrXnDeU9DqKF9LWrC+mL+PX8L7czbQNCGOK07owM9O7ESLJgoEkVBTEESThe/B27/0lqk45R7o/3Nv6WsfLdqwnUfHL+H9OetJio/l8hOyueakTrRUIIiEjIIg2uwohHd+BYvegw4nwrmPQYsOflfF4o3b+fv4pbw7ex2N42O5bEAHrj2pE6lNE/wuTSTiKQiikXMw87/wwe3e5yNGw9E/CflNaLVZWuAFwtuz1pEYF8vlAzpwzcmdSFMgiASNgiCabV0Fb94Iq76AnJFw1iPQNN3vqgBYWrCDf4xfwtuz1pEQF8ul/bO49uTOpCcrEETqm4Ig2lVWwtePwWe/g4RkOPtR6D7S76r2WV64g3+MX8qbM9fSKC6GnxzfgesGdSIjOdHv0kQihoJAPBvnwxvXwoY5cPSlMPxBSGzmd1X7rNi0c18gxMUYlxyfxQ2DOpPRTIEgcqQUBFKlfA98/hB88TdolgnnPQ7ZJ/pd1fes3LSTf05Yyrjv1hIbY1zSL4vrB3WmdYoCQeRwKQhkf2umwhvXwZYVMODnMPQer29yGFm9eRf/nLCUsTPyiYkxLj6uPTcM7kyblMZ+lybS4CgIpHZ7dsLH98C0ZyG9B5z/JLTp7XdV+1mzZRePTVzK69PyiTHjwuMyuXFwF9o2VyCI1JWCQA5uyafw1s9h12YYfAcMvBliw6/pjBcIyxgzfQ0AF/Ztz41DutBOgSDyg3wJAjN7DjgTKHDO9axluwGPAGcAu4ArnXMzfuh9FQRBsmsLvHcLzHsDMvvBeU9Aame/q6rV2m27eWzCUl6b5gXCqD7tuXFwZ9q3TPK5MpHw5VcQnAzsAF44QBCcAfwCLwiOBx5xzh3/Q++rIAiyOWO8QKgog9MfgGOv8LqihaF123bz+MRlvPrtGiqdY1SfTH4+pIsCQaQWvp0aMrNs4N0DBMGTwETn3MuBzxcBg51z6w/2ngqCEChe550qWjYektK8tph5F0LbY8PizuSa1hft5omJy3j52zVUVjrOP7YdNw3pSlaqAkFkr3ANgneB0c65LwKffwbc7pzb77e8mV0LXAuQlZXVZ9WqVUGrWQKcg0UfwOxXvP9W7IHUrtD7Iuh1YVisXVTThqISnvh8Gf+dupqKSsd5x7TjpiFdyE5r4ndpIr4L1yB4D3iwRhDc5pybfrD31IjAB7u3wfw3YfZrsOpL77msE7xRQu650LiFv/XVUFBcwhOfL+c/36yivNJx7tHtuGloFzoqECSKhWsQ6NRQQ7R1Fcx5DWa9CpuXQGwj6DYc8i6Crqf72hCnpoLtJTwZCIQ95ZX7AqFTuj8tPUX8FK5BMBK4iaqLxY86536wz6KCIEw4B+u+80YJc8fAzkJvZJB7vhcK7fuFzfWEwu2lPDVpGS9+7QXC2b3bctPQrnTJUCBI9PBr1tDLwGAgDdgI/BaIB3DOPRGYPvoPYDje9NGrars+UJOCIAxVlMGyCTD7Va8xTvluaNHRC4S8C8NmGuqmHaU8PWk5L0xZRUl5BWflteWXp3ShS0ay36WJBJ1uKJPQKSmGBe94obBiEuAg8zgvFHLPhyapflfI5h2lPD15BS9MWcnusgpG9mrDL0/pSrdWCgSJXAoC8UfRWpjzuhcKBfMhJg66nObNPOo2wve1jbbs3MMzk5fz769WsqusgjN6eoGQ01qBIJFHQSD+2zDHC4TZr8OODZCQAkedDb0v9mYg+dhXeevOPTz7xQr+9dVKdpSWM6Jna355Sld6tAmfJbpFjpSCQMJHZQWs+Ny7yDz/bSjbCSntodePvFBIz/GttG27AoHw5Uq2l5YzPNcLhKPaKhCk4VMQSHjas9O7uDz7Ve8uZlcJbY72rif0GgVNM3wpq2hXGc9+uYLnv1zB9pJyTj+qFb88pSs926X4Uo9IfVAQSPjbvhHmjvXuZF4/CywWOg+BvIu9tpqNQr9cRNHuMp7/cgXPfbGC4pJyTu3Ril+d0pVemQoEaXgUBNKwFCwMXE94DYrzoVFT6HGWN1LoeDLExIa0nOKSMv715Uqembyc4pJyTumewa9O7UpeZvOQ1iFyJBQE0jBVVsLqr2DWKzD/LSgthuQ2gUXwLobW+92nGFTFJWX8+8uVPPPFCop2lzEkJ51fndqNo9srECT8KQik4SvbDYs/9Ja2WPoJVJZDq57eDWu9fgTN2oaslO0lZbwwZRVPT17Otl1lDOqWzt4/mBIAABCPSURBVK9O7cqxWeG15pJIdQoCiSw7N3kNdGa9AmunAeadMup9sXcKKSE09wHsKC3nhSkreXrScrbuKuOkrmmM6pPJoG7pNE8KnzWXREBBIJFs87LA9YRXYetKiGvsXVzufTF0GhKSlps7S8t58etVPDN5BZt2lBJjcGxWC4Z0z2BITgY92iRjYbLukkQvBYFEPudgzVRv1tHccVCyDZqkQ89R3p3MbY4O+iJ4FZWO2fnbmLCwgPGLCpi7thiANimJDM7JYGj3DAZ2SSWpUfj1g5bIpyCQ6FJeCks+8UJh8UdeU520HO96Qt6F0DwrJGUUFJcwcVEh4xcWMHlJITv3VNAoNobjO7VkaHcvGDqkqkeChIaCQKLX7q0w703v1NHqKd5zHQZ6U1GPOgcah2bGz57ySr5duYXxCwuYsKiA5YU7AeiU3oShORkM6Z7BcdktaRTn31IbEtkUBCLgXUOY/bo3Uti8FGITIGe4NxW1y6khbaqzctNOJiwqYPzCAr5ZvoU9FZU0TYjjxC5pDO2eweDu6WQk+7son0QWBYFIdc7BuhneVNS5Y2HXJmjcEnqe74VCZt+QNtXZWVrOl0s3MWFRIRMWFrChuASAXu1SAhec0+md2ZyYGF1wlsOnIBA5kIoyb52jWa/AovehvARadqpqqtOyU0jLcc6xYP32faOF71ZvpdJBapNGDMpJZ2j3DE7qmk5K4/iQ1iUNn4JApC5KimHB214orPwCr6lOP2/WUe75kNQy5CVt3bmHSUu8C86fLy5k264yYmOMPh1a7Lvg3DWjqaanyg9SEIgcqqJ8r6nOrFehcAHExEPX071Q6DrMl6Y6FZWO71ZvDYwWClmw3pue2q55Y4Z2z2BI93RO6JxGYnxo12KShkFBIHK4nKtqqjPnddixERJT4KhzvdNHWQN8a6qzvmg3ExZ6o4Uvl25id1kFCXExnNA5NRAMGWS2CP2qrRKeFAQi9aGyApZP9EJhwTtQtgtSsgL3J1wE6d18K62krIKpK6qmp67avAuAbq2aMiQwPbVPhxbEx2p6arRSEIjUt9IdgaY6r3jhsLepTpdTvVFC+36Q6E9nM+ccyzftZEIgFKau2EJZhSM5MY6Tu6UzNCeDQTnppDVN8KU+8YeCQCSYtm+AOWO8hfDWfQeuAiwGWuV6/Ziz+kOHEyC5tT/llZTx5dJNgdFCIYXbSzGDvMzmDA0sfZHbtpmmp0Y4BYFIqJTu8FZEXTXFu5M5/1vvFBJAi+zvB0Nql5DerwBQWemYv76Y8Qu96amz8rfhHKQnJzC4mzc99cSuaSQnanpqpFEQiPilogw2zK4KhtVTYNdmb1tSmhcKWQOgwwBo3Tskq6VWt2lHKZ8vKmT8ogImLS5ke0k58bHGcdkt911w7pTWRNNTI4CCQCRcOOctb7HqK1j9tdeBbetKb1t8E++u5g6BUUPmcdAodIvSlVdUMn3VVsYvKmDCwgIWb9wBQFbLpH2hcHzHlpqe2kApCETCWfH6qtHC6imwYS7gwGKhTe+qYMgaAE3SQlZW/tZd+5a9+HLpJkrLK2kcH8vAwHpIQ7qn0yalccjqkSOjIBBpSEqKYM233mhh9deQPw0qSr1tqV2900hZgUeL7JBcZygpq2DKss37ri2s3bYbgO6tk/fd4XxMVgtidcE5bCkIRBqy8lJYN7MqGFZP8cICILlN1Wgha4A3UykmuKdunHMsLdixLxSmrdpKRaWjeVI8g7qlMyQng0Hd0mnRRO06w4mCQCSSVFZC4cKqYFg1BYrzvW0Jzbx7GPYGQ7s+QV8Oo2h3GZOXFDJhYSETFxWweeceYgyOyfLWQ1K7zvCgIBCJdNvWVF1jWDXFWx8JILYRtD2mKhiyjofGLYJWRmWlY/baIu+ehYUFzFnrjVxaN0tkSHdvtDCwSxpNEtSuM9QUBCLRZtcWWPNNVTCs+w4qy7xtGUdVBUOHAZCSGbQyCrZ77TonLCxg8pJN7Cgt39euc0iO15Utp3WyOrOFgIJAJNqV7Ya106uCYc1U2LPd25bSPhAMgRvd0nKCspDenvJKpq3csq/XwrJAu85GsTHktE6mZ7sUerVLIS8zhW6tFA71zbcgMLPhwCNALPCMc250je0pwEtAFhAH/MU59/zB3lNBIFIPKitg49zANYavvIDYsdHb1rgFtO9fFQxtjg5KG881W3YxK38bc9YWMXdtEXPyiyguKQcUDsHgSxCYWSywGDgNyAe+BX7snJtfbZ+7gBTn3O1mlg4sAlo75/Yc6H0VBCJB4BxsXfH9YNi81NsWlwjt+gaCYYDXrCcIC+o551izZTez1yocgsGvIBgA3OecGxb4/E4A59yD1fa5E2gP/BzIBj4BujnnKg/0vgoCkRDZURi4AB24A3r97GoL6vWsusaQNSBoC+opHOqPX0EwChjunPtZ4PPLgOOdczdV2ycZeBvoDiQDFznn3qvlva4FrgXIysrqs2rVqqDULCIHUbrDW0RvbzDkT6u2oF7HandAnwCpnYN2o5vC4fD4FQQ/AobVCIJ+zrlfVNtnFDAQuAXojDci6O2cKz7Q+2pEIBImKsq8UcLqWhbUa5L+/RvdWucFdUE9hcMPO1gQBHMybz7eaZ+9MoF1Nfa5ChjtvDRaamYr8EYHU4NYl4jUh9h4yOzjPU64ybvOsGnJ94NhwTvevvFNoP1xVcGQ2bdeF9QzM7JSk8hKTeLMvLZA7eHw3ux1vDx1NaBwqC6YI4I4vIvFpwBr8S4WX+Kcm1dtn8eBjc65+8ysFTADb0Sw6UDvqxGBSANSvK5qWYzqC+rFxHkL6rXOg4wekN7du7+haXpQy4nmkYOf00fPAB7Gmz76nHPuD2Z2PYBz7gkzawv8C2gDGN7o4KWDvaeCQKQB+96Cet94U1hLtlVtT0r1AiG9O2R0r/o4qWXQSoqWcNANZSISnpzz7l8oWOCtn1QwHwoWep/vveENoGmrwMihR7WAyIHElCCVFXnhoCAQkYbFOSheGwiF+YGQCITF3plKAM3aVTu11MN7pOVAQtMglNSww0FBICKRobISilbXCIj5ULi4qmcDQPOsaqeY9gZEN4iv30Y6DSkcFAQiEtkqK7yWn3tPLRUu8EYQm5ZULbZnMV4jn5oBkdoF4hLqrZRwDQcFgYhEp4oy2LJ8/4DYvMy7Sxq8lqCpXbxrD+k9qgKiZSdvimw9CIdwUBCIiFRXXuqtpVSwoOpRuAC2rAACvxNj4r3TSTUDokV2vXSBC3U4KAhEROpizy7YtLjq4vTegNi2umqfuMRAQFS7/yGjO6RkHfHy3T8UDleekM19Z+ce1nsrCEREjkTpDihcVHVqae8MpuK1VfvEN/GmtNYMiGbtjmjdpb3hMGdtEe1bNiYvs/lhvY+CQEQkGHZvqz0g9vZ2AK+P9N4b5KqfYmraKmgL89XGr7WGREQiW+PmXh/orOO///yuLVWnlQoWeBeqF7wLM16o2iexedWooXpANEkL7feAgkBEpP4ltYTsgd5jL+dgZ+H3rz0ULIQ5Y6G0qNpr06pCofoppsYtglaugkBEJBTMoGmG9+g0qOp552D7+v0DYuZ/Yc+Oqv2atvZWeT3hF/u/9xFSEIiI+MkMmrX1Hl1OqXreOSha8/27qJPbBKUEBYGISDgy85bKaJ4F3U4P6pcKj9WQRETENwoCEZEopyAQEYlyCgIRkSinIBARiXIKAhGRKKcgEBGJcgoCEZEo1+BWHzWzQmDVYb48DdhUj+XUl3CtC8K3NtV1aFTXoYnEujo459Jr29DgguBImNm0Ay3D6qdwrQvCtzbVdWhU16GJtrp0akhEJMopCEREoly0BcFTfhdwAOFaF4Rvbarr0KiuQxNVdUXVNQIREdlftI0IRESkBgWBiEiUi8ggMLPhZrbIzJaa2R21bDczezSwfbaZHRsmdQ02syIzmxl43Buiup4zswIzm3uA7X4drx+qK+THy8zam9kEM1tgZvPM7Fe17BPy41XHuvw4XolmNtXMZgXqur+Wffw4XnWpy5efx8DXjjWz78zs3Vq21f/xcs5F1AOIBZYBnYBGwCzgqBr7nAF8ABjQH/gmTOoaDLzrwzE7GTgWmHuA7SE/XnWsK+THC2gDHBv4OBlYHCb/vupSlx/Hy4CmgY/jgW+A/mFwvOpSly8/j4GvfQvw39q+fjCOVySOCPoBS51zy51ze4BXgHNq7HMO8ILzfA00N7PgNAM9tLp84ZybBGw5yC5+HK+61BVyzrn1zrkZgY+3AwuAdjV2C/nxqmNdIRc4Bns7sMcHHjVnqPhxvOpSly/MLBMYCTxzgF3q/XhFYhC0A9ZU+zyf/X8g6rKPH3UBDAgMVz8ws9wg11RXfhyvuvLteJlZNnAM3l+T1fl6vA5SF/hwvAKnOWYCBcAnzrmwOF51qAv8+ff1MHAbUHmA7fV+vCIxCKyW52omfV32qW91+Zoz8NYD6Q38HXgzyDXVlR/Hqy58O15m1hQYC9zsnCuuubmWl4TkeP1AXb4cL+dchXPuaCAT6GdmPWvs4svxqkNdIT9eZnYmUOCcm36w3Wp57oiOVyQGQT7QvtrnmcC6w9gn5HU554r3Dledc+8D8WaWFuS66sKP4/WD/DpeZhaP98v2P865cbXs4svx+qG6/P735ZzbBkwEhtfY5Ou/rwPV5dPxGgicbWYr8U4fDzWzl2rsU+/HKxKD4Fugq5l1NLNGwMXA2zX2eRu4PHD1vT9Q5Jxb73ddZtbazCzwcT+8/z+bg1xXXfhxvH6QH8cr8PWeBRY45/52gN1CfrzqUpdPxyvdzJoHPm4MnAosrLGbH8frB+vy43g55+50zmU657LxfkeMd85dWmO3ej9ecUfy4nDknCs3s5uAj/Bm6jznnJtnZtcHtj8BvI935X0psAu4KkzqGgXcYGblwG7gYheYJhBMZvYy3gyJNDPLB36Ld/HMt+NVx7r8OF4DgcuAOYHzywB3AVnV6vLjeNWlLj+OVxvg32YWi/eL9DXn3Lt+/zzWsS5ffh5rE+zjpSUmRESiXCSeGhIRkUOgIBARiXIKAhGRKKcgEBGJcgoCEZEopyAQCSHzVrTcb0VJET8pCEREopyCQKQWZnapeevVzzSzJwMLlO0ws7+a2Qwz+8zM0gP7Hm1mX5u3NvwbZtYi8HwXM/s0sGjZDDPrHHj7pmY2xswWmtl/9t69KuIXBYFIDWbWA7gIGBhYlKwC+AnQBJjhnDsW+BzvTmeAF4DbnXN5wJxqz/8H+Gdg0bITgL3LABwD3AwchdefYmDQvymRg4i4JSZE6sEpQB/g28Af643xliquBF4N7PMSMM7MUoDmzrnPA8//G3jdzJKBds65NwCccyUAgfeb6pzLD3w+E8gGvgj+tyVSOwWByP4M+Ldz7s7vPWl2T439DrY+y8FO95RW+7gC/RyKz3RqSGR/nwGjzCwDwMxamlkHvJ+XUYF9LgG+cM4VAVvN7KTA85cBnwd6AeSb2bmB90gws6SQfhcidaS/RERqcM7NN7O7gY/NLAYoA34O7ARyzWw6UIR3HQHgCuCJwC/65VStBnkZ8KSZ/S7wHj8K4bchUmdafVSkjsxsh3Ouqd91iNQ3nRoSEYlyGhGIiEQ5jQhERKKcgkBEJMopCEREopyCQEQkyikIRESi3P8D6qE9vt0ODpMAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxU5fX48c8hBLJDAiEkJGGXVUSIuCACLggC2tbWfV/QWvu1m1W/rWtrv/5qF6t1qVpc6lbrUhEQEQVRUVkUFQJIWBMCBBIgCUnIdn5/3JtkCBOYYCZ3Mjnv1yuvzMy9d+bMJcyZ5znP81xRVYwxxpjGOngdgDHGmNBkCcIYY4xfliCMMcb4ZQnCGGOMX5YgjDHG+GUJwhhjjF+WIEy7JyJ9RERFpGMA+14lIh+3RlzGeM0ShGlTRGSziFSKSPdGj690P+T7eBPZQbHEikipiMz1OhZjvgtLEKYt2gRcXHdHRI4For0L5xA/BA4Ak0QktTVfOJBWkDGBsgRh2qJ/AVf43L8SeN53BxHpIiLPi8guEdkiIr8VkQ7utggR+ZOI7BaRjcBUP8f+U0S2i8g2Efm9iEQ0I74rgSeAr4FLGz13hoi84cZVKCJ/99l2vYisEZESEckWkVHu4yoiA3z2e1ZEfu/eniAieSJym4jsAJ4RkUQRme2+xh73drrP8Uki8oyI5Lvb/+s+vkpEpvvsF+meo5HNeO8mjFiCMG3RZ0CCiAxxP7gvBF5otM8jQBegHzAeJ6Fc7W67HpgGHA9k4Xzj9/UcUA0McPeZBFwXSGAikglMAF50f67w2RYBzAa2AH2AXsAr7rYfAfe4+ycA5wKFgbwm0BNIAnoDM3D+Xz/j3s8EyoG/++z/LyAGGAb0AP7qPv48cJnPfucA21V1ZYBxmHCjqvZjP23mB9gMnAn8Fvg/YDLwHtARUJwP3gicLp6hPsfdACxyb38A3OizbZJ7bEcgxT022mf7xcBC9/ZVwMeHie+3wEr3dhpQAxzv3j8Z2AV09HPcu8AtTTynAgN87j8L/N69PQGoBKIOE9NIYI97OxWoBRL97JcGlAAJ7v3XgF97/W9uP979WH+laav+BSwG+tKoewnoDnTC+aZeZwvON3ZwPghzG22r0xuIBLaLSN1jHRrtfzhXAE8BqGq+iHyI0+X0JZABbFHVaj/HZQAbAnyNxnapakXdHRGJwWkVTAYS3Yfj3RZMBlCkqnsaP4kb7yfA+SLyJjAFuOUoYzJhwLqYTJukqltwitXnAG802rwbqML5sK+TCWxzb2/H+aD03VYnF6cF0V1Vu7o/Cao67EgxicgpwEDgDhHZ4dYETgQudovHuUBmE4XkXKB/E09dhtMlVKdno+2Nl2T+JTAIOFFVE4DT6kJ0XydJRLo28VrP4XQz/Qj4VFW3NbGfaQcsQZi27FrgdFXd7/ugqtYArwL3i0i8iPQGfkFDneJV4H9EJF1EEoHbfY7dDswH/iwiCSLSQUT6i8j4AOK5Eqe7ayhOt85IYDjOh/sUYClOcnrAHQobJSJj3WOfBn4lIqPFMcCNG2AlcIlbXJ+MU1M5nHicusNeEUkC7m70/t4BHnOL2ZEicprPsf8FRuG0HBq3zEw7YwnCtFmqukFVlzex+afAfmAj8DHwEjDT3fYUTp//V8AXHNoCuQKniyob2IPTF3/Y4aoiEgVcADyiqjt8fjbhdIdd6Sau6TjF761AHk6BHVX9D3C/G2cJzgd1kvv0t7jH7cUZFfXfw8UCPIQz7Hc3TkF/XqPtl+O0sNYCBcDP6jaoajnwOk7XXePzYtoZUbULBhljGojIXcAxqnrZEXc2Yc2K1MaYem6X1LU4rQzTzlkXkzEGcCbq4RSx31HVxV7HY7xnXUzGGGP8shaEMcYYv8KqBtG9e3ft06eP12EYY0ybsWLFit2qmuxvW1gliD59+rB8eVOjHo0xxjQmIlua2mZdTMYYY/yyBGGMMcYvSxDGGGP8CqsahD9VVVXk5eVRUVFx5J3bsKioKNLT04mMjPQ6FGNMmAj7BJGXl0d8fDx9+vTBZ/nmsKKqFBYWkpeXR9++fb0OxxgTJsK+i6miooJu3bqFbXIAEBG6desW9q0kY0zrCvsEAYR1cqjTHt6jMaZ1hX0XkzHGhJXqSijdAcXboXgblGyHmko49ect/lKWIIJs7969vPTSS9x0003NOu6cc87hpZdeomvXpi78ZYwJOwdKoTgfSvKd33U/JW4yKN4O+wsOPS4uxRJEW7R3714ee+yxQxJETU0NERERTR43d+7cYIdmjGktqlBW2PAhX58AfFoBxflwoPjQY6MTIT4NEtIg9ThI6AXxqc7vhFTn8ajgfJG0BBFkt99+Oxs2bGDkyJFERkYSFxdHamoqK1euJDs7m+9973vk5uZSUVHBLbfcwowZM4CGZUNKS0uZMmUKp556KkuWLKFXr1689dZbREdHe/zOjDEA1FRByY6Dv+X7fujXtQBqKg8+Tjo43/wT0qDbAOg73v3Ar0sAac7vTg2XI6+pVXYWV5BbVMbWojJyN5eTW7SFmtrNPHzx8S3+1tpVgrj37dVk5/vJ0N/B0LQE7p7e9PXsH3jgAVatWsXKlStZtGgRU6dOZdWqVfXDUWfOnElSUhLl5eWccMIJnH/++XTr1u2g51i/fj0vv/wyTz31FBdccAGvv/46l11mF/syJugq9/vv5vHtBiotABpdNqFjVMO3/Iwx7oe92wqo++CPS4GIQz+C95VXOQkgv4zcou1OIthTTm5RGdv2lFNZU1u/bweB1C7RDEyJC8rbb1cJIhSMGTPmoLkKDz/8MG+++SYAubm5rF+//pAE0bdvX0aOHAnA6NGj2bx5c6vFa0xYUoWyIj99/b5dP/lwYN+hx0Z1afiWnzLc50PfJwFEJ0ITIwsPVNewbU85uRv2sLWojLy61sCeMrYWllFcUX3Q/okxkWQkxTA0LYGzh/UkMymGjKRoMpNiSOsaTWRE8AajtqsEcbhv+q0lNja2/vaiRYtYsGABn376KTExMUyYMMHvXIbOnTvX346IiKC8vLxVYjWmTaqpbhjlc7hib82BRgeKT5dPf+g7rqGrpz4BpEKnWL8vW0dV2VVywOdDv7z+dm5RGTuKK/C9Tlunjh3ISIwmIymGUZmJZCbFkJ4YU58I4qO8Wx2hXSUIL8THx1NSUuJ32759+0hMTCQmJoa1a9fy2WeftXJ0xrQxlWWH7+svzndG+WjtwcdFdGr4kO+VBUNSDy32xqVARGAfxiUVVeQWldd/6NfXBNyuoAPVDa8vAinxUWQmxXBK/+713/4zkpwkkBzXmQ4dQnMekyWIIOvWrRtjx45l+PDhREdHk5KSUr9t8uTJPPHEE4wYMYJBgwZx0kkneRipMSGgqhzyV8Ludf4TQMXeQ4/p3MX9lp8KKUMP7etP6AUxSU12+fgNo6aW7XsrGloBRQ2JIHdPOUX7Dy44x0d1JDMphgHJcZw+uEd9iyAjKYZeXaOJimx6xGIoC6trUmdlZWnjCwatWbOGIUOGeBRR62pP79WEiX15kPs55C6DvKWw/WuorXI3CsT1OHRIZ+ME0Ln5BVpVpXB/Zf03/7w95WwtbKgFbN9XQU1tw2djZITQq2vDh35mUgwZbjdQZlIMXWLa7iKZIrJCVbP8bbMWhDGmdVQfcBJA3tKGpFCS72zrGA29RsHJP4GME6HncOfDP8AuH3/KK2vqC78NrYBytxVQRlllzUH7J8d3JiMxmqzebh2gLhEkxdAzIYqIEO0GCiZLEMaY4CjZ4SaCpZC3zOk6qisMd8mE3qc4Q0DTT4CexzY7GdTUKtv3lR/0ob+1vh5Qzu7Sg4vQMZ0inG/83WIYO6A7mUnR9a2B9MQYoju1zW6gYLIEYYz57mqqYMc3TiKoax3s2+psi+gMaSNhzPVO6yBjDMT3POJTqir7yqvqv/lvre8Ocn7n7y2nqqahGyiig5DWNYqMxBjOHNKjvjsoI9EpCifFdrJFLZvJEoQxpvlKd7ldRW7rYNsXUO0Ov45Pc5LASTdC+hhIHQEdOx/26XaVHCB7ezHZ+cVkby9mQ0EpuUVllBw4eE5AUmwnMpJiOLZXF6Yem1rfAshMiqFnl6igzgloj4KaIERkMvA3IAJ4WlUfaLS9C/ACkOnG8idVfSaQY40xraSmGgqyGxJC7lLYs8nZ1iHSSQCjr3KSQsYY6JLe5FPV1iqbC/cflAyy84spKGnoDkpPjGZgjzhO6JPYUBB2f+I623fa1hS0sy0iEcCjwFlAHrBMRGaparbPbj8BslV1uogkA+tE5EWgJoBjjTHBUFbkdhUtdZJC3gqo2u9si+3hJIGsq53WQdpIiPS/LlhFVQ3rdpQclAzWbC+uLw537CAMTIln3MBkhqYlMDTV+WnLI4LCTTDT8RggR1U3AojIK8B5gO+HvALx4nQMxgFFQDVwYgDHtglHu9w3wEMPPcSMGTOIiYk58s7GHI3aWti19uDWQeF6Z5tEOKOJRl7i1g5OgK69/c4nKNpf6SaBfWTnF7M6v5gNu0qpGyka37kjQ9ISuCAroz4ZDEyJo3NHKwyHsmAmiF5Ars/9PJwPfl9/B2YB+UA8cKGq1opIIMcCICIzgBkAmZmZLRN5C2pque9APPTQQ1x22WWWIEzLqdgHecsPbh3UrTcU081pFYy82Pnda9Qhy0rU1iq5RfvrWwSr853WwY7ihiVi0rpEMTQtgSnDezI0LYFhaV1IT4y2AnEbFMwE4e+vofGsvLOBlcDpQH/gPRH5KMBjnQdVnwSeBGei3FFHGyS+y32fddZZ9OjRg1dffZUDBw7w/e9/n3vvvZf9+/dzwQUXkJeXR01NDXfeeSc7d+4kPz+fiRMn0r17dxYuXOj1WzFtjSrsXn9w62DXWkCdpaZ7DIVjz3eSQcYYSOp3UOvgQHUN67fVtQj2uV1EJZS6heOIDsKA5DhO7t+NoakJDEtLYEhqAomxnTx6w6alBTNB5AEZPvfTcVoKvq4GHlBnOneOiGwCBgd4bPO9c7szFK8l9TwWpjRdP/dd7nv+/Pm89tprLF26FFXl3HPPZfHixezatYu0tDTmzJkDOGs0denShb/85S8sXLiQ7t27t2zMJjwdKIVtK3xaB8ugfI+zLaqrM99g+A+cZNBrNHSOrz90b1kl2RsLnZaB2zrIKSil2u0jiu0UwZDUBH4wqhfD0hIYmtqFgSlxbXYJCROYYCaIZcBAEekLbAMuAi5ptM9W4AzgIxFJAQYBG4G9ARzb5syfP5/58+dz/PHOhT1KS0tZv34948aN41e/+hW33XYb06ZNY9y4cR5HakKeKhRtbCgm5y6FgtUNi9QlD4Yh0xtaB90GQocOqCp5e8pZvb6Y7O3byc53Csfb9jasEJyS0JmhqQmcOSSlvl6QmRQTsgvKmeAJWoJQ1WoRuRl4F2eo6kxVXS0iN7rbnwB+BzwrIt/gdCvdpqq7Afwd+52DOsw3/dagqtxxxx3ccMMNh2xbsWIFc+fO5Y477mDSpEncddddHkRoQlZlGeR/6UxCq0sKZbudbZ3iIT0LTrvVSQjpoyE6kcrqWtYXlJC9pZjsz9bUtwxK3OsNdBDolxxHVp9ErkjtzVC3i6h73OHnLJj2I6iDilV1LjC30WNP+NzOByYFemxb5Lvc99lnn82dd97JpZdeSlxcHNu2bSMyMpLq6mqSkpK47LLLiIuL49lnnz3oWOtiamdUYe9Wn9bB57BzFdS6k8a6DYCBkxrmHSQPZt+BWtbUDSdduZXs/FWsLyipn2kcHRnBkNR4zhuZxtDULgxNS2BQSrwtL2EOy2adBJnvct9Tpkzhkksu4eSTTwYgLi6OF154gZycHG699VY6dOhAZGQkjz/+OAAzZsxgypQppKamWpE6nFVVwPav3NbBUmeZitIdzrbIGKdeMPYWSB+DpmeRXxXbUCtYs4/V+R+St6ehiyg53ukiGj8o2ZlbkJZAn26x7XKxOfPd2HLfYaQ9vdc2bd+2hkSQ+7mTHOqWuE7sU183qEo7gQ0dMsneUdYw63h7MXvLnH1FoG/32PokUPe7R3yUd+/NtDm23LcxXqmphh1fwVaf1kFxnrOtYxSkjYKTb6I8ZTRrIwfz9Z7OzpDSz4v5dsd2Kmu2AdC5YwcGpyYwZXhqfTIYkhpPTCf7L2yCx/66jGlJ1ZWQ/wVs+QQ2f+K0ECpLnW1dMtCMMZQk38i6yMEsLevFqp3lZK8sZkthGZADOAvSDUtL4OqxfdyJZk4XUUdbiM60snaRIFQ17GdxhlNXYZtSVe4Uk7csgc0fO7er3VnFyUPguItYF3Us75b0ZenuKLLXFlO0ohKoBDbRp1sMw9IS+NHodIaldXG7iDqH/d+raRvCPkFERUVRWFhIt27dwvY/napSWFhIVJT1PQfdgRKnVbBlidNC2LbCrR+IM2ky6xrnQjiZp7CxrDP3z1nD+2sL6NSxjEEpEZw1JIVhvZwuosGpCbY6qQlpYf/XmZ6eTl5eHrt27fI6lKCKiooiPb3pZZbNUSrfA1s/c1oHW5Y4BWWtcRaySzseTr4Jeo91FrOL7grAvvIqHn5/Pc8t2UxUZAR3TBnMlaf0sVnHps0J+wQRGRlJ3759vQ7DtBWlu2Cr2zrYssSZf4BCRCfolQXjfuG0ENLHQOe4gw6tqVVeXrqVv7z3LXvKKrkwK4NfThpEcrxNPDNtU9gnCGMOqzi/oX6wZQnsXuc83jHamYQ28X+dhNArCyKb7sJbkrOb+2Zns3ZHCWP6JnHXtKEM79Wlld6EMcFhCcK0H6qwd0tD/WDLJw1XRusUD5knOUtd9x4LqSOh45FXJd1SuJ/756xhfvZO0hOjeezSUUwZ3jNs612mfbEEYcKXKhTmNAw53bKkYQ5CdKKTCMZc7/zueSx0CLxGUFJRxd8/yOGZTzbTMUK49exBXHtqX6szmLBiCcKEj9pa2LXm4C6j/QXOttge0Gcs9P6ZkxCSB0OH5s8rqKlV/rM8lz/NX8fu0kp+ODqdX589iB4JNoLMhB9LEKbtqqmGnd80dBltXdJw/YOEdOg/0akf9D4VuvX3e6nM5vh8YyH3zc5mdX4xWb0TmXnVCYxI79oCb8SY0GQJwrQdNVXOktd1XUZbP4NKZ6VcEvvC4KlO66D3WOia+Z0TQp3cojL+7501zP1mB2ldonj44uOZPiLV6gwm7FmCMKGrqgK2LT94lnJVmbOt+yAY8SM3IZwCCWkt/vL7D1Tz2KIcnvpoExEi/PzMY5hxWj9bItu0G5YgTOg4UOosaFdXUN62HGoqAYGU4TDqivpZysQlBy2M2lrljS+38cd5aykoOcD3RqZx25TBpHaJDtprGhOKLEEY75TvdZatqJ+lvNK5KI5EQOpxcOINTgsh8yRn1FErWL65iPtmZ/N13j5GZnTlictHMyqzdV7bmFBjCcK0nv2FPrOUP4Ed3wAKHSKdS2aOvcVpIWScCJ3jWzW0bXvLeeCdtbz9VT49E6L464XHcd5xvew6zKZdswRhgqdkx8FzEHatcR7vGA0ZJ8CE250WQnoWRHrTfVNWWc0TH27kycUbUIX/OX0AN07ob9dZMAZLEKYl7d168ByEog3O453inG6iET9yhpymHR/QLOVgUlXeWpnPA++sZUdxBdNGpHL7lMGkJ8Z4GpcxocQShDk6qlC08eAWwr6tzraork5XUdbV7izlERAROn9qK3P3cu/bq/ly616O7dWFRy45nhP6JHkdljEhJ3T+15q2obIMFtwD2W9B6Q7nsdhkJyGccrOTEHoMPapZysG2Y18Ff5y3lje+3EZyfGce/OEIzh+VbnUGY5pgCcIErnADvHoF7FwNw74HfU9zuoy6D2yxSWnBUFFVw5OLN/L4og3UqHLThP7cNHGAXazHmCOw/yEmMGvnwps3Oong0tdg4JleR3REqsrsr7fzwDtr2ba3nCnDe/K/5wwhI8nqDMYEwhKEObzaGlh4P3z0Z2duwgX/gsTeXkd1RN/k7eO+2atZtnkPQ1IT+PMFx3FSv25eh2VMm2IJwjRtfyG8fg1sXATHXw7n/OmwF80JBQUlFTw4bx2vfZFHt9hOPPCDY/lRVgYRVmcwptksQRj/8lY49Yb9u+DcR5xlLkJYRVUNMz/ZxKMf5FBZU8uMcf34yekDSIiK9Do0Y9osSxDmYKqwfCbMux3ie8K17zrzFkKUqvLu6h3cP3cNuUXlnDU0hd+cM4Q+3WO9Ds2YNi+oCUJEJgN/AyKAp1X1gUbbbwUu9YllCJCsqkUishkoAWqAalXNCmasBmcI65xfwlcvwYAz4QdPQUzozg9Ynb+P383O5rONRQxKiefF605k7IDuXodlTNgIWoIQkQjgUeAsIA9YJiKzVDW7bh9VfRB40N1/OvBzVS3yeZqJqro7WDEaH0Ub4d9XwM5VMP52GH9bSM5lANhdeoA/z1/HK8ty6Rodye+/N5yLTsigY0RoxmtMWxXMFsQYIEdVNwKIyCvAeUB2E/tfDLwcxHhMU9bNgzdmuENY/wMDz/I6Ir8qq2t5dskmHnk/h/KqGq4Z25f/OWMgXaKtzmBMMAQzQfQCcn3u5wEn+ttRRGKAycDNPg8rMF9EFPiHqj7ZxLEzgBkAmZmZLRB2O1JbAwv/AB/9yR3C+jwk9vE6qkOoKgvWFHD/nGw2F5Zx+uAe/GbqEPonx3kdmjFhLZgJwt+4Qm1i3+nAJ426l8aqar6I9ADeE5G1qrr4kCd0EseTAFlZWU09v2lsfyG8fi1sXBjSQ1jX7Sjhd7Oz+ThnNwN6xPHcNWMYf0zwLhZkjGkQzASRB2T43E8H8pvY9yIadS+par77u0BE3sTpsjokQZijsG0FvHollBbA9Idh9JVeR3SIov2V/OW9dbz0+VbioyK5Z/pQLj2pN5FWZzCm1QQzQSwDBopIX2AbThK4pPFOItIFGA9c5vNYLNBBVUvc25OA+4IYa/ugCiuegXdug7jQHMJaVVPL859u4W8LvmV/ZQ2Xn9Sbn515DImx3i4Pbkx7FLQEoarVInIz8C7OMNeZqrpaRG50tz/h7vp9YL6q7vc5PAV4U5wF4DoCL6nqvGDF2i5UlcPsX4T0ENaFawv43ZxsNu7az7iB3blz2lCOSWndK8sZYxqIavh022dlZeny5cu9DiP0FG2CVy93LvE5/nYY/2voEOF1VPVyCkr43ew1fPjtLvp1j+W304YwcVAPJIRXiDUmXIjIiqbmmdlM6nC3bh68OcO5fcl/4JhJ3sbjY29ZJQ8tWM+/PttCTKcIfjt1CFec3IdOHa3OYEwosAQRrmprYNEDsPiP0PNYZxXWpL5eRwVAdU0tL36+lb8u+Jbi8iouHpPJL846hm5xnb0OzRjjwxJEOCorcoawbvgARl4GU/8EkdFeRwXA4m938bvZ2awvKOWU/t24c9pQhqQmeB2WMcYPSxDhZtsXziqspTth+t9g1JUhcbW3jbtKuX/OGt5fW0DvbjH84/LRTBqaYnUGY0KYJYhwoQpfPAdzb4W4FLjmXeg1yuuo2FdexSPvr+e5TzfTuWMEd0wZzFVj+9C5Y+gUyY0x/lmCCAdV5TDnV7DyBeh/OvzgaYj19uppNbXKK8u28uf537KnrJILszL45aRBJMdbncGYtsISRFtXtMnpUtrxNZz2a5hwu+dDWJfk7Oa+2dms3VHCmL5J3DVtKMN7dfE0JmNM81mCaMu+nQ9vXOfcvuRVOOZsT8PZUrif++esYX72TtITo3ns0lFMGd7T6gzGtFGWINqi2hr48P85PyEwhLWkooq/L8zhmY830zFCuPXsQVx7al+iIq3OYExbZgmirSkrgtevgw3vw8hLYeqfPRvCWlOrvLYilwff/ZbdpQc4f1Q6v548iJSE0FsV1hjTfJYg2pJtX7irsO6AaQ/B6Ks8G8K6dFMR9769mtX5xWT1TmTmVVmMSO/qSSzGmOCwBNFWrHgO5v7KHcI6D3qN9iyU37z5DS9+vpW0LlE8fPHxTB+RanUGY8KQJYhQV1XuJIYvQ2MI64Lsnbz4+VauPLk3t08ZQnQnqzMYE64sQYSyPZvh35e7Q1hvhQl3eDqEtaKqhnveXs0xKXH8dtpQu3iPMWHOEkSoWv+eU4xG4eJ/w6DJXkfE44s2kLennJevP8mSgzHtgCWIUFNb2zCENWU4XBgaq7BuKdzP4x9u4Nzj0ji5v7eztI0xrcMSRCgpK4I3roecBXDcJc4Q1k4xXkcFwH1vZxPZQfjN1CFeh2KMaSWWIEJF/krnqm8lO2DaX2H01SGxCis4hen31xbwm3OG2BwHY9oRSxCh4IvnncX2YpPh6nmQ7t0Q1sYqqmq4d/ZqBvaI46qxfbwOxxjTiixBeKmqwh3C+i/oNwHOn+n5KqyNPfHhBnKLrDBtTHsU0P94EXldRKaKiH1CtJQ9W2DmJCc5jPsVXPZGyCWHrYVlPLZoA9OtMG1MuxToB/7jwCXAehF5QEQGBzGm8Ld+ATw5Hoo2w8WvwBl3er5Etz/3zV7tFKbPscK0Me1RQAlCVReo6qXAKGAz8J6ILBGRq0UkMpgBhpXaWlj0ALz4Q0joBTMWwqApXkfl1/trdrJgTQG3nDmQnl2sMG1MexRwDUJEugGXAZcDXwIvAqcCVwITghFcWCkrgjdmQM57cNzFMPUvITOEtbG6GdMDe8Rx9Vjv52AYY7wRUIIQkTeAwcC/gOmqut3d9G8RWR6s4MJG3RDW4u1OYsi6JmSGsPpTV5h+6foTrTBtTDsWaAvi76r6gb8NqprVgvGEny/+BXN+CbHdnVVY00P7dG0tLONxtzB9Sv/uXodjjPFQoF8Ph4hI/WL/IpIoIjcFKabwUFUBs34Ks26G3ifDDYtDPjmAU5juaIVpYwyBJ4jrVXVv3R1V3QNcf6SDRGSyiKwTkRwRud3P9ltFZKX7s0pEakQkKZBjQ9rerTDzbGcC3LhfukNYQ//buBWmjTG+Au1i6iAioqoKICIRQKfDHeDu8yhwFpAHLBORWaqaXbePqj4IPOjuPx34uaoWBXJsyMpZ4KzCWlsLF70Mg8/xOqKAVFTVcO/b2QywwrQxxhVoC+Jd4FUROYL5jNoAABSHSURBVENETgdeBuYd4ZgxQI6qblTVSuAV4LzD7H+x+7xHc6z3amvhwz/CCz5DWNtIcgD4x4cb2VpUxn3nDrPCtDEGCLwFcRtwA/BjQID5wNNHOKYXkOtzPw840d+OIhIDTAZuPopjZwAzADIzM48QUpCU74E3boD178KIi5zF9kJ0CKs/uUVlPLYoh2kjUjllQOh3hRljWkdACUJVa3FmUz/ejOf2N45Tm9h3OvCJqhY191hVfRJ4EiArK6up5w+e7V85V30rzneW5866NqSHsPpz79vZRHQQfjt1qNehGGNCSKDzIAYC/wcMBeqrl6ra7zCH5QEZPvfTgfwm9r2Ihu6l5h7rnS9fcIawxnRrE0NY/flg7U4WrNnJHVMGW2HaGHOQQDubn8FpPVQDE4HncSbNHc4yYKCI9BWRTjhJYFbjnUSkCzAeeKu5x3qmqgLevgXe+glkjGkzQ1gbq6iq4Z5Z2fRPjrXCtDHmEIEmiGhVfR8QVd2iqvcApx/uAFWtxqkpvAusAV5V1dUicqOI3Oiz6/eB+aq6/0jHBvqmgmrvVnhmMqx4Fk79OVz2ZpsYwurPk4vdwvR5w+nU0QrTxpiDBVqkrnCX+l4vIjcD24AeRzpIVecCcxs99kSj+88CzwZyrOdy3ofXr4XaGrjoJRg81euIjlpuURmPLsxh6ohUxlph2hjjR6BfG38GxAD/A4zGWbTvymAFFXLqh7CeD/FpMGNRm04OAPfNritM24xpY4x/R2xBuJPWLlDVW4FS4OqgRxVKDhrCeqE7hDXW66i+k4VrC3gveye3TxlMapdor8MxxoSoIyYIVa0RkdG+M6nbje1fO6uw7tsG5/wJTriuzQ1hbaxuKe/+ybFcY4VpY8xhBFqD+BJ4S0T+A/gWk98ISlSh4MsXYc4vIDoJrn4HMk7wOqIW8eTijWwpLOPF6060wrQx5rACTRBJQCEHj1xSIPwSRPUBeOc2WPEM9D0Nzp8JccleR9UirDBtjGmOQGdSt4+6w95cePUKyP/CGcI68bcQEfBF90KeFaaNMc0R6EzqZ/Cz1IWqXtPiEXllwwfw2rVQWw0XvghDpnkdUYtauM4K08aY5gn06/Fsn9tROJPbQm/pi6NRWwsf/xk+uB96DIELX4Bu/b2OqkU5M6atMG2MaZ5Au5he970vIi8DC4ISUWs7sA+WPwvH/gimP9Tmh7D685RbmH7hWitMG2MCd7Qd7AMBj9bWbmHRic61G2KT2/wQVn9yi8p4dFEOU49N5dSBVpg2xgQu0BpECQfXIHbgXCMiPMQdcdWQNut3s7MRhN9YYdoY00yBdjHFBzsQ0/IWritgfvZObps8mLSuVpg2xjRPQB3SIvJ9d1nuuvtdReR7wQvLfFcHqmu4d9Zq+iXHcu2pVpg2xjRfoBXLu1V1X90dVd0L3B2ckExLeGrxRjYXlnHvucOsMG2MOSqBfnL42y98ZpCFmbw9Zfx9YQ7nHNuTcQPDYxa4Mab1BZoglovIX0Skv4j0E5G/AiuCGZg5enWFabvGtDHmuwg0QfwUqAT+DbwKlAM/CVZQ5ugtWlfAu6t38tMzBlhh2hjznQQ6imk/cHuQYzHf0YFqZ8Z0v+RYrju1n9fhGGPauEBHMb0nIl197ieKyLvBC8scDStMG2NaUqCfIt3dkUsAqOoeArgmtWk9dYXpKcOtMG2MaRmBJohaEalfWkNE+uBndVfjnd/PXuMUpqdZYdoY0zICHar6G+BjEfnQvX8aMCM4IZnm+vDbXcxbvYNbzx5ELytMG2NaSKBF6nkikoWTFFYCb+GMZDIeqy9Md4/lunE2Y9oY03ICXazvOuAWIB0nQZwEfMrBlyA1Hnj6o01s2r2f568ZQ+eOEV6HY4wJI4HWIG4BTgC2qOpE4HhgV9CiMgHZtrecRz5Yz5ThPTntGCtMG2NaVqAJokJVKwBEpLOqrgUGBS8sE4jfvZ1thWljTNAEWqTOc+dB/Bd4T0T2EC6XHG2jrDBtjAm2QIvU33dv3iMiC4EuwLygRWUOq64w3dcK08aYIGr2dFtV/VBVZ6lq5ZH2FZHJIrJORHJExO9SHSIyQURWishqn2G0iMhmEfnG3ba8uXGGs7rC9D3nDrPCtDEmaIK2ZLeIRACPAmcBecAyEZmlqtk++3QFHgMmq+pWEWk8O3uiqu4OVoxtUV1hevKwnoy3wrQxJoiCuWDPGCBHVTe6rY1XgPMa7XMJ8IaqbgVQ1YIgxhMWfj/bya93TrfCtDEmuIKZIHoBuT7389zHfB0DJIrIIhFZISJX+GxTYL77eJOztkVkhogsF5Hlu3aF98jbxd/u4p1VO/jp6QOtMG2MCbpgXhVO/DzWeP2mjsBo4AwgGvhURD5T1W+Bsaqa73Y7vScia1V18SFPqPok8CRAVlZW2K4PZYVpY0xrC2YLIg/I8LmfzqFDY/OAeaq63601LAaOA1DVfPd3AfAmTpdVu/XPjzexcfd+7p4+1ArTxphWEcwEsQwYKCJ9RaQTcBEwq9E+bwHjRKSjiMQAJwJrRCRWROIBRCQWmASsCmKsIW3b3nIeeT+Hs4elMGGQrbJujGkdQetiUtVqEbkZeBeIAGaq6moRudHd/oSqrhGRecDXQC3wtKquEpF+wJsiUhfjS6rabudd3D8nG0W502ZMG2NaUTBrEKjqXGBuo8eeaHT/QeDBRo9txO1qau8+Wr+Lud/s4FeTjiE9McbrcIwx7YhdlzKEHaiu4e63VtOnWwzXn2bXmDbGtK6gtiDMd1NXmH726hOsMG2MaXXWgghR+VaYNsZ4zBJEiPq9FaaNMR6zBBGC6grTN08cYIVpY4xnLEGEmMrqWu6eZYVpY4z3rEgdYv758SY27trPM1aYNsZ4zFoQISTfXcp70tAUJlph2hjjMUsQIeT+OWuoqbXCtDEmNFiCCBEfr9/NnG+2c/PEAWQkWWHaGOM9SxAhoLK6lrtmraK3FaaNMSHEitQhYOYnDYXpqEgrTBtjQoO1IDy2fV85D7+/nrOsMG2MCTGWIDz2e7cwfZcVpo0xIcYShIc+Xr+bOV9v5ydWmDbGhCBLEB5xZkw7hekZVpg2xoQgK1J7ZOYnm9iwaz/PXGWFaWNMaLIWhAfqCtNnDklh4mArTBtjQpMlCA/UzZi+e7oVpo0xocsSRCv7JGc3s7/ezk0TrDBtjAltliBaUd1S3plJMdww3grTxpjQZkXqVvTMJ5vIKShl5lVZVpg2xoQ8a0G0ku37yvmbW5g+fXCK1+EYY8wRWYJoJVaYNsa0NZYgWsESK0wbY9ogSxBB5izlbYVpY0zbY0XqIHt2iRWmjTFtU1BbECIyWUTWiUiOiNzexD4TRGSliKwWkQ+bc2yo27GvgocWrOfMIT2sMG2MaXOC1oIQkQjgUeAsIA9YJiKzVDXbZ5+uwGPAZFXdKiI9Aj22Lbh/bl1hepjXoRhjTLMFswUxBshR1Y2qWgm8ApzXaJ9LgDdUdSuAqhY049iQtmTDbt7+Kp8fT+hvhWljTJsUzATRC8j1uZ/nPubrGCBRRBaJyAoRuaIZx4asqppa7n5rNRlJ0dw4vr/X4RhjzFEJZpFa/Dymfl5/NHAGEA18KiKfBXis8yIiM4AZAJmZmUcdbEt69pPNrC8o5Z9XWmHaGNN2BbMFkQdk+NxPB/L97DNPVfer6m5gMXBcgMcCoKpPqmqWqmYlJye3WPBHa2dxBQ8t+JYzBvfgjCFWmDbGtF3BTBDLgIEi0ldEOgEXAbMa7fMWME5EOopIDHAisCbAY0PS/XPWUGWFaWNMGAhaF5OqVovIzcC7QAQwU1VXi8iN7vYnVHWNiMwDvgZqgadVdRWAv2ODFWtLWbJhN7O+yueWMwaS2c0K08aYtk1U/Xbtt0lZWVm6fPlyT167qqaWc/72ERXVNbz38/FWezDGtAkiskJVs/xts6U2WkhdYfruacMsORhjwoIliBbgW5g+c6gVpo0x4cESRAuwwrQxJhxZgviOPt1QyKyv8vnx+P5WmDbGhBVLEN9BVU0td721ioykaH48wWZMG2PCiyWI7+C5JU5h+i4rTBtjwpAliKO0s7iCv773LacP7sGZQ3p4HY4xxrQ4SxBH6Q9z6wrTQxHxt3SUMca0bZYgjsJnGwt5a2U+N47vT+9usV6HY4wxQWEJopnqCtPpidHcZIVpY0wYswTRTM8t2cy3O0u5e7oVpo0x4c0SRDMUFDvXmJ44KNkK08aYsGcJohn+MHcNlTW13HPuMCtMG2PCniWIAH22sZD/rsznxtP6WWHaGNMuWIIIQN01ptMTo/nxhAFeh2OMMa3CEkQAnluymXU7S7hr2lCiO1lh2hjTPliCOIK6wvSEQcmcZUt5G2PaEUsQR/CHuWuorK7lnulWmDbGtC+WIA7jc7cwfcP4fvTpboVpY0z7YgmiCc6M6dX06hrNTVaYNsa0Q5YgmvD8p1ucwvR0K0wbY9onSxB+FBRX8NB73zJhUDKTrDBtjGmnLEH48X/vrOWAFaaNMe2cJYhGPt9YyJtfbrPCtDGm3bME4aO6ppa7Z1lh2hhjwBLEQZ7/dAtrd1hh2hhjwBJEvYIS5xrT44+xwrQxxoAliHoPzHUL07aUtzHGAEFOECIyWUTWiUiOiNzuZ/sEEdknIivdn7t8tm0WkW/cx5cHM86lm4p448ttzDitH32tMG2MMQB0DNYTi0gE8ChwFpAHLBORWaqa3WjXj1R1WhNPM1FVdwcrRnAK03e9tYpeXaP5yUQrTBtjTJ2gJQhgDJCjqhsBROQV4DygcYLwVEV1LSPSu3D64BQrTBtjjI9gdjH1AnJ97ue5jzV2soh8JSLviMgwn8cVmC8iK0RkRlMvIiIzRGS5iCzftWtXs4OM69yRP/7wOCYP79nsY40xJpwFswXhr9Krje5/AfRW1VIROQf4LzDQ3TZWVfNFpAfwnoisVdXFhzyh6pPAkwBZWVmNn98YY8xRCmYLIg/I8LmfDuT77qCqxapa6t6eC0SKSHf3fr77uwB4E6fLyhhjTCsJZoJYBgwUkb4i0gm4CJjlu4OI9BR3TKmIjHHjKRSRWBGJdx+PBSYBq4IYqzHGmEaC1sWkqtUicjPwLhABzFTV1SJyo7v9CeCHwI9FpBooBy5SVRWRFOBNN3d0BF5S1XnBitUYY8yhRDV8uu2zsrJ0+fKgTpkwxpiwIiIrVDXL3zabSW2MMcYvSxDGGGP8sgRhjDHGr7CqQYjILmDLUR7eHQjqsh5HyeJqHoureSyu5gnHuHqrarK/DWGVIL4LEVneVKHGSxZX81hczWNxNU97i8u6mIwxxvhlCcIYY4xfliAaPOl1AE2wuJrH4moei6t52lVcVoMwxhjjl7UgjDHG+GUJwhhjjF/tKkEEcI1sEZGH3e1fi8ioEImryWt3BzmumSJSICJ+V9L18HwdKS6vzleGiCwUkTUislpEbvGzT6ufswDjavVzJiJRIrLUvWDYahG5188+XpyvQOLy5G/Mfe0IEflSRGb72day50tV28UPzoqyG4B+QCfgK2Boo33OAd7BudjRScDnIRLXBGC2B+fsNGAUsKqJ7a1+vgKMy6vzlQqMcm/HA9+GyN9YIHG1+jlzz0GcezsS+Bw4KQTOVyBxefI35r72L4CX/L1+S5+v9tSCqL9GtqpWAnXXyPZ1HvC8Oj4DuopIagjE5Ql1ruBXdJhdvDhfgcTlCVXdrqpfuLdLgDUcepndVj9nAcbV6txzUOrejXR/Go+a8eJ8BRKXJ0QkHZgKPN3ELi16vtpTggjkGtmBXke7teOCpq/d7SUvzlegPD1fItIHOB7n26cvT8/ZYeICD86Z212yEigA3lPVkDhfAcQF3vyNPQT8GqhtYnuLnq/2lCACuUZ2IPu0tOZcu/s44BGca3eHAi/OVyA8PV8iEge8DvxMVYsbb/ZzSKucsyPE5ck5U9UaVR2Jc0niMSIyvNEunpyvAOJq9fMlItOAAlVdcbjd/Dx21OerPSWII14jO8B9Wj0uPcy1uz3mxfk6Ii/Pl4hE4nwIv6iqb/jZxZNzdqS4vP4bU9W9wCJgcqNNnv6NNRWXR+drLHCuiGzG6Yo+XUReaLRPi56v9pQgjniNbPf+Fe5IgJOAfaq63eu4pIlrdwc5rkB4cb6OyKvz5b7mP4E1qvqXJnZr9XMWSFxenDMRSRaRru7taOBMYG2j3bw4X0eMy4vzpap3qGq6qvbB+Zz4QFUva7Rbi56voF2TOtRoYNfInoszCiAHKAOuDpG4/F67O9ixicjLOKM1uotIHnA3TsHOs/MVYFyenC+cb3iXA9+4/dcA/wtk+sTmxTkLJC4vzlkq8JyIROB8wL6qqrO9/j8ZYFxe/Y0dIpjny5baMMYY41d76mIyxhjTDJYgjDHG+GUJwhhjjF+WIIwxxvhlCcIYY4xfliCMCQHirA56yOqcxnjJEoQxxhi/LEEY0wwicpk41wpYKSL/cBd1KxWRP4vIFyLyvogku/uOFJHPxFmX/00RSXQfHyAiC9yF3r4Qkf7u08eJyGsislZEXqybqWuMVyxBGBMgERkCXAiMdRdyqwEuBWKBL1R1FPAhzsxugOeB21R1BPCNz+MvAo+6C72dAtQthXA88DNgKM71QcYG/U0ZcxjtZqkNY1rAGcBoYJn75T4aZznoWuDf7j4vAG+ISBegq6p+6D7+HPAfEYkHeqnqmwCqWgHgPt9SVc1z768E+gAfB/9tGeOfJQhjAifAc6p6x0EPitzZaL/DrV9zuG6jAz63a7D/n8Zj1sVkTODeB34oIj0ARCRJRHrj/D/6obvPJcDHqroP2CMi49zHLwc+dK/DkCci33Ofo7OIxLTquzAmQPYNxZgAqWq2iPwWmC8iHYAq4CfAfmCYiKwA9uHUKQCuBJ5wE8BGGlbWvBz4h4jc5z7Hj1rxbRgTMFvN1ZjvSERKVTXO6ziMaWnWxWSMMcYva0EYY4zxy1oQxhhj/LIEYYwxxi9LEMYYY/yyBGGMMcYvSxDGGGP8+v8s1Lq+VdgA2QAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plot_loss(model_0_history)\n", "plot_acc(model_0_history)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**QUESTIONS**:\n", "\n", "1. What is the effect of changing the learning rate?\n", " (Examples: 0.03, 0.003, or 0.00003)\n", " \n", "2. What will happen if we increase the `epochs` value? (To 10, 20?)\n", "\n", "3. What is the ultimate accuracy of one-layer model compared to Decision Tree and Logistic Regression?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4.2 One Hidden Layer\n", "\n", "Apparently, the first NN model that we created above did not perform very well.\n", "One way to improve the performance of a NN model is to add one or more hidden layers.\n", "The function below has a hidden layer, an output layer, and utilizes the `adam` optimizer that was used in the previous notebook.\n", "We will use this function to test the performance based on different parameters (number of hidden neurons, hidden layers, learning rate, etc.).\n", "To start, let us try an example of a model with 1 hidden layer, `18` hidden neurons, and a learning rate of `0.0003`." ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [], "source": [ "def NN_Model(hidden_neurons,learning_rate):\n", " \"\"\"Definition of deep learning model with one dense hidden layer\"\"\"\n", " model = Sequential([\n", " # More hidden layers can be added here\n", " Dense(hidden_neurons, activation='relu',input_shape=(19,),kernel_initializer='random_normal'), # Hidden Layer\n", " Dense(18, activation='softmax') # Output Layer\n", " ])\n", " adam=tf.keras.optimizers.Adam(lr=learning_rate, beta_1=0.9, beta_2=0.999, amsgrad=False)\n", " model.compile(optimizer=adam,\n", " loss='categorical_crossentropy',\n", " metrics=['accuracy'])\n", " return model" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Train on 218461 samples, validate on 54616 samples\n", "Epoch 1/10\n", " - 11s - loss: 1.0628 - acc: 0.7179 - val_loss: 0.5014 - val_acc: 0.8793\n", "Epoch 2/10\n", " - 10s - loss: 0.3785 - acc: 0.9135 - val_loss: 0.3049 - val_acc: 0.9296\n", "Epoch 3/10\n", " - 10s - loss: 0.2662 - acc: 0.9365 - val_loss: 0.2355 - val_acc: 0.9412\n", "Epoch 4/10\n", " - 10s - loss: 0.2144 - acc: 0.9466 - val_loss: 0.1942 - val_acc: 0.9555\n", "Epoch 5/10\n", " - 10s - loss: 0.1805 - acc: 0.9579 - val_loss: 0.1662 - val_acc: 0.9597\n", "Epoch 6/10\n", " - 10s - loss: 0.1554 - acc: 0.9640 - val_loss: 0.1446 - val_acc: 0.9654\n", "Epoch 7/10\n", " - 10s - loss: 0.1366 - acc: 0.9682 - val_loss: 0.1284 - val_acc: 0.9689\n", "Epoch 8/10\n", " - 10s - loss: 0.1223 - acc: 0.9713 - val_loss: 0.1164 - val_acc: 0.9714\n", "Epoch 9/10\n", " - 10s - loss: 0.1104 - acc: 0.9737 - val_loss: 0.1048 - val_acc: 0.9739\n", "Epoch 10/10\n", " - 10s - loss: 0.1003 - acc: 0.9756 - val_loss: 0.0958 - val_acc: 0.9745\n" ] } ], "source": [ "model_1 = NN_Model(18,0.0003)\n", "model_1_history=model_1.fit(Rec.train_features,\n", " Rec.train_L_onehot,\n", " epochs=10, batch_size=32,\n", " validation_data=(Rec.test_features, Rec.test_L_onehot),\n", " verbose=2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Self Exploration:**\n", "\n", "Now that we know how to use the function `NN_model`, we can use it to run a variety of tests using different parameters.\n", "Below is a list of what could be interesting in exploring; feel free to experiment with your own ideas as well.\n", "\n", "- Test with different number of neurons in the hidden layer: **25, 36, 40, 80**\n", " - It is also worthwhile to test a lower number of neurons: **12, 8, 4, 2, 1**\n", "- Test with different learning rate: **0.001, 0.01, 0.1**\n", "- Test with different batch size: **16, 32, 64, 128, 512, 1024**\n", "- Test with different number of hidden layers: **2,3,4...**\n", "\n", "**NOTE:**\n", "The easiest way to do this exploration is to simply copy the code in the cell above and paste it in a new cell below, since most of the parameters (`hidden_neurons`, `learning_rate`, `batch_size`, etc.) can be changed when calling the function or when fitting the model.\n", "However, to change the number of hidden layers, the original function `NN_model` will need to be modified, therefore, it is best to do this last." ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [], "source": [ "\"\"\"Start self exploration here\"\"\";\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Remarks\n", "\n", "This process of experimentation with different parameters for the neural network can get repetitive and cause this notebook to become very long.\n", "Instead, it would be more beneficial to run experiments like this in a scripting environment.\n", "To do this, we need to identify the relevant code elements for our script.\n", "In a general sense, this is what we should pick out:\n", "\n", "* Useful Python libraries & user-defined functions\n", "* Proper sequence of commands that were run throughout this notebook (i.e. one-hot encoding must be done before training the models)\n", "* Code cells that require repetition to run many tests (i.e. the cells right above this section)\n", "\n", "In brief, once the initial experiments are done and we have established a working pipeline for machine learning, we need to change the way we work.\n", "Real machine-learning work requires many repetitive experiments, each of which may take a long time to complete.\n", "Instead of running many experiments in Jupyter notebooks, where each will require us to wait for a while to finish), we need to be able to carry out many experiments in parallel so that we can obtain our results in a timely manner.\n", "This is key reason why we really should make a script for these experiments and submit the script to run them in batch (non-interactive model).\n", "HPC is well suited for this type of workflow--in fact it is most efficient when used in this way.\n", "Here are the key components of the \"batch\" way of working:\n", "\n", "* A job scheduler (such as SLURM job scheduler on HPC) to manage our jobs and run them on the appropriate resources;\n", "* The machine learning script written in Python, which will read inputs from files and write outputs to files and/or standard output;\n", "* The job script to launch the machine learning script in the non-interactive environment (e.g. HPC compute node);\n", "* A way to systematically repeat the experiments with some variations. This can be done by adding some command-line arguments for the (hyper)parameters that will be varied for each test.\n", "\n", "In your hands-on package, there is a folder called `expts-sherlock` which contains a sample Python script and SLURM job script that you can submit to the HPC cluster:\n", "\n", "* `NN_Model-064n.py` shows an example of how a script converted from this notebook would look like.\n", " We recommend only one experiment per script to avoid complication.\n", "\n", "* `NN_Model-064n.wahab.job` is the corresponding job script for ODU's Wahab cluster." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.3" } }, "nbformat": 4, "nbformat_minor": 4 }