![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Data Analysis"
+ ],
+ "metadata": {
+ "id": "31CSwzTOcdHd"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# imports \n",
+ "from google.colab import drive\n",
+ "import json\n",
+ "import shutil\n",
+ "import os\n",
+ "import zipfile\n",
+ "import warnings\n",
+ "from datetime import datetime, timedelta\n",
+ "import pandas as pd\n",
+ "import numpy as np\n",
+ "import seaborn as sns \n",
+ "import statistics\n",
+ "from collections import defaultdict, Counter\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd"
+ ],
+ "metadata": {
+ "id": "lrTEvPO0OhI_"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import src.framework__data_set as ds\n",
+ "import random\n",
+ "random.seed(10)"
+ ],
+ "metadata": {
+ "id": "Xqarye5kabvu"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Data Understanding\n",
+ "Let's start our analysis by examining the data and assessing the applications. While we have three metrics available (Container memory, Container CPU, and Node memory), we'll focus on only two of them (Container memory, Container CPU). Therefore, we'll begin by evaluating the usage data for Container memory."
+ ],
+ "metadata": {
+ "collapsed": false,
+ "id": "3c2rxhluplZE"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Number of applications: 417\n",
+ "Example of 15 applications, and number of samples for each: [('kube-rbac-proxy', 7718487), ('cni-plugins', 3142572), ('driver-registrar', 2820695), ('oauth-proxy', 1942223), ('collector', 1701727), ('nmstate-handler', 1587755), ('keepalived', 1587373), ('node-exporter', 1586814), ('coredns', 1586715), ('whereabouts-cni-bincopy', 1586041), ('bridge-marker', 1583131), ('whereabouts-cni', 1580344), ('egress-router-binary-copy', 1578966), ('kube-multus-additional-cni-plugins', 1577492), ('coredns-monitor', 1576538)]\n"
+ ]
+ }
+ ],
+ "source": [
+ "hist = ds.get_amount_of_data_per_application(\n",
+ " metric=\"container_mem\",\n",
+ " path_to_data=\"./data/\"\n",
+ ")\n",
+ "print(\"Number of applications: \",len(hist))\n",
+ "print(\"Example of 15 applications, and number of samples for each: \",hist[:15])"
+ ],
+ "metadata": {
+ "id": "-7tcL5G1plZE",
+ "outputId": "aafe498d-464b-48bb-8065-5380e280ebe0"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Number of applications: 378\n",
+ "Example of 15 applications, and number of samples for each: [('kube-rbac-proxy', 7578821), ('oauth-proxy', 1885203), ('collector', 1602125), ('coredns-monitor', 1564425), ('network-metrics-daemon', 1559549), ('nmstate-handler', 1556575), ('dns-node-resolver', 1554940), ('dns', 1554337), ('node-ca', 1552028), ('kube-multus-additional-cni-plugins', 1549615), ('network-check-target-container', 1547871), ('cni-plugins', 1546571), ('keepalived', 1545578), ('coredns', 1545393), ('node-exporter', 1545075)]\n"
+ ]
+ }
+ ],
+ "source": [
+ "hist = ds.get_amount_of_data_per_application(\n",
+ " metric=\"container_cpu\",\n",
+ " path_to_data=\"./data/\"\n",
+ ")\n",
+ "print(\"Number of applications: \",len(hist))\n",
+ "print(\"Example of 15 applications, and number of samples for each: \",hist[:15])"
+ ],
+ "metadata": {
+ "id": "ycHeEfpxplZG",
+ "outputId": "0ac4a472-87f3-4b3a-fb75-8e0e27757820"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Data Visualization\n",
+ "We can gain insights into the data by analyzing the graphs and their trends. As an illustration, consider the two plots below of the 'bridge-maker' application with container CPU information. To generate these plots, we utilized the 'get_data_set' function, which accepts 'metric' and 'application_name' as inputs and extracts the corresponding data from the json files. The output is a dataset object comprising a list of pandas dataframes with two columns - one for time and one for the sample."
+ ],
+ "metadata": {
+ "collapsed": false,
+ "id": "78CjPODDplZH"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [
+ {
+ "data": {
+ "text/plain": "| \n", + " | application | \n", + "namespace | \n", + "node | \n", + "pod | \n", + "start | \n", + "stop | \n", + "data | \n", + "
|---|---|---|---|---|---|---|---|
| 0 | \n", + "ads-ui | \n", + "api-designer | \n", + "oct-03-17-compute | \n", + "ads-ui-30-kxgl5 | \n", + "2022-05-28 17:00:00 | \n", + "2022-05-29 01:59:00 | \n", + "[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... | \n", + "
| 1 | \n", + "ads-ui | \n", + "api-designer | \n", + "oct-03-17-compute | \n", + "ads-ui-30-kxgl5 | \n", + "2022-05-29 03:01:00 | \n", + "2022-05-30 03:04:00 | \n", + "[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... | \n", + "
| 2 | \n", + "ads-ui | \n", + "api-designer | \n", + "oct-03-17-compute | \n", + "ads-ui-30-kxgl5 | \n", + "2022-05-30 05:01:00 | \n", + "2022-05-31 01:04:00 | \n", + "[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... | \n", + "
| 3 | \n", + "ads-ui | \n", + "api-designer | \n", + "oct-03-17-compute | \n", + "ads-ui-30-kxgl5 | \n", + "2022-05-31 03:01:00 | \n", + "2022-05-31 09:04:00 | \n", + "[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... | \n", + "
| 4 | \n", + "ads-ui | \n", + "api-designer | \n", + "oct-03-17-compute | \n", + "ads-ui-30-kxgl5 | \n", + "2022-05-31 10:00:00 | \n", + "2022-05-31 10:59:00 | \n", + "[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... | \n", + "
| \n", + " | application | \n", + "namespace | \n", + "node | \n", + "pod | \n", + "time | \n", + "median | \n", + "min | \n", + "max | \n", + "mean | \n", + "rolling_average | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "alertmanager | \n", + "openshift-monitoring | \n", + "oct-03-03-compute | \n", + "alertmanager-main-0 | \n", + "2022-06-03 05:01:00 | \n", + "0.007690 | \n", + "0.001593 | \n", + "0.008225 | \n", + "0.007221 | \n", + "0.007221 | \n", + "
| 1 | \n", + "alertmanager | \n", + "openshift-monitoring | \n", + "oct-03-03-compute | \n", + "alertmanager-main-0 | \n", + "2022-06-03 05:31:00 | \n", + "0.007784 | \n", + "0.007409 | \n", + "0.008186 | \n", + "0.007779 | \n", + "0.007779 | \n", + "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Data Analysis"
+ ],
+ "metadata": {
+ "id": "31CSwzTOcdHd"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# imports \n",
+ "from google.colab import drive\n",
+ "import json\n",
+ "import shutil\n",
+ "import os\n",
+ "import zipfile\n",
+ "import warnings\n",
+ "from datetime import datetime, timedelta\n",
+ "import pandas as pd\n",
+ "import numpy as np\n",
+ "import seaborn as sns \n",
+ "import statistics\n",
+ "from collections import defaultdict, Counter\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd\n"
+ ],
+ "metadata": {
+ "id": "lrTEvPO0OhI_"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "import src.framework__data_set as ds\n",
+ "import random\n",
+ "random.seed(10)"
+ ],
+ "metadata": {
+ "id": "PYSzwTyYplZD"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Data Understanding\n",
+ "Let's start our analysis by examining the data and assessing the applications. While we have three metrics available (Container memory, Container CPU, and Node memory), we'll focus on only two of them (Container memory, Container CPU). Therefore, we'll begin by evaluating the usage data for Container memory."
+ ],
+ "metadata": {
+ "collapsed": false,
+ "id": "3c2rxhluplZE"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Number of applications: 417\n",
+ "Example of 15 applications, and number of samples for each: [('kube-rbac-proxy', 7718487), ('cni-plugins', 3142572), ('driver-registrar', 2820695), ('oauth-proxy', 1942223), ('collector', 1701727), ('nmstate-handler', 1587755), ('keepalived', 1587373), ('node-exporter', 1586814), ('coredns', 1586715), ('whereabouts-cni-bincopy', 1586041), ('bridge-marker', 1583131), ('whereabouts-cni', 1580344), ('egress-router-binary-copy', 1578966), ('kube-multus-additional-cni-plugins', 1577492), ('coredns-monitor', 1576538)]\n"
+ ]
+ }
+ ],
+ "source": [
+ "hist = ds.get_amount_of_data_per_application(\n",
+ " metric=\"container_mem\",\n",
+ " path_to_data=\"./data/\"\n",
+ ")\n",
+ "print(\"Number of applications: \",len(hist))\n",
+ "print(\"Example of 15 applications, and number of samples for each: \",hist[:15])"
+ ],
+ "metadata": {
+ "id": "-7tcL5G1plZE",
+ "outputId": "aafe498d-464b-48bb-8065-5380e280ebe0"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Number of applications: 378\n",
+ "Example of 15 applications, and number of samples for each: [('kube-rbac-proxy', 7578821), ('oauth-proxy', 1885203), ('collector', 1602125), ('coredns-monitor', 1564425), ('network-metrics-daemon', 1559549), ('nmstate-handler', 1556575), ('dns-node-resolver', 1554940), ('dns', 1554337), ('node-ca', 1552028), ('kube-multus-additional-cni-plugins', 1549615), ('network-check-target-container', 1547871), ('cni-plugins', 1546571), ('keepalived', 1545578), ('coredns', 1545393), ('node-exporter', 1545075)]\n"
+ ]
+ }
+ ],
+ "source": [
+ "hist = ds.get_amount_of_data_per_application(\n",
+ " metric=\"container_cpu\",\n",
+ " path_to_data=\"./data/\"\n",
+ ")\n",
+ "print(\"Number of applications: \",len(hist))\n",
+ "print(\"Example of 15 applications, and number of samples for each: \",hist[:15])"
+ ],
+ "metadata": {
+ "id": "ycHeEfpxplZG",
+ "outputId": "0ac4a472-87f3-4b3a-fb75-8e0e27757820"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Data Visualization\n",
+ "We can gain insights into the data by analyzing the graphs and their trends. As an illustration, consider the two plots below of the 'bridge-maker' application with container CPU information. To generate these plots, we utilized the 'get_data_set' function, which accepts 'metric' and 'application_name' as inputs and extracts the corresponding data from the json files. The output is a dataset object comprising a list of pandas dataframes with two columns - one for time and one for the sample."
+ ],
+ "metadata": {
+ "collapsed": false,
+ "id": "78CjPODDplZH"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [
+ {
+ "data": {
+ "text/plain": "| \n", + " | application | \n", + "namespace | \n", + "node | \n", + "pod | \n", + "start | \n", + "stop | \n", + "data | \n", + "
|---|---|---|---|---|---|---|---|
| 0 | \n", + "collector | \n", + "openshift-logging | \n", + "oct-03-14-compute | \n", + "collector-45ggj | \n", + "2022-05-28 17:00:00 | \n", + "2022-05-28 17:04:00 | \n", + "[0.0320022991333341, 0.0262894255359178, 0.020... | \n", + "
| 1 | \n", + "collector | \n", + "openshift-logging | \n", + "oct-03-14-compute | \n", + "collector-45ggj | \n", + "2022-05-28 21:01:00 | \n", + "2022-05-29 03:04:00 | \n", + "[0.0091724990056144, 0.0154374696894289, 0.021... | \n", + "
| 2 | \n", + "collector | \n", + "openshift-logging | \n", + "oct-03-14-compute | \n", + "collector-45ggj | \n", + "2022-05-29 05:01:00 | \n", + "2022-05-30 03:04:00 | \n", + "[0.006033915309428, 0.0122509814599761, 0.0192... | \n", + "
| 3 | \n", + "collector | \n", + "openshift-logging | \n", + "oct-03-14-compute | \n", + "collector-45ggj | \n", + "2022-05-30 05:01:00 | \n", + "2022-05-30 19:04:00 | \n", + "[0.0126865943562504, 0.0207211524625931, 0.029... | \n", + "
| 4 | \n", + "collector | \n", + "openshift-logging | \n", + "oct-03-14-compute | \n", + "collector-45ggj | \n", + "2022-05-30 21:01:00 | \n", + "2022-05-31 01:04:00 | \n", + "[0.0067857535893381, 0.0126782464223269, 0.019... | \n", + "
| \n", + " | application | \n", + "namespace | \n", + "node | \n", + "pod | \n", + "time | \n", + "median | \n", + "min | \n", + "max | \n", + "mean | \n", + "rolling_average | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "collector | \n", + "openshift-logging | \n", + "oct-03-07-compute | \n", + "collector-48kdf | \n", + "2022-06-07 01:01:00 | \n", + "0.284603 | \n", + "0.066266 | \n", + "0.349879 | \n", + "0.265529 | \n", + "0.265529 | \n", + "
| 1 | \n", + "collector | \n", + "openshift-logging | \n", + "oct-03-07-compute | \n", + "collector-48kdf | \n", + "2022-06-07 01:31:00 | \n", + "0.289773 | \n", + "0.046554 | \n", + "0.370590 | \n", + "0.275501 | \n", + "0.275501 | \n", + "
| \n", + " | application | \n", + "namespace | \n", + "node | \n", + "pod | \n", + "time | \n", + "median | \n", + "min | \n", + "max | \n", + "mean | \n", + "rolling_average | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "dns | \n", + "openshift-dns | \n", + "oct-03-14-compute | \n", + "dns-default-6hbkp | \n", + "2022-06-08 13:01:00 | \n", + "0.004850 | \n", + "0.001546 | \n", + "0.005141 | \n", + "0.004609 | \n", + "0.004609 | \n", + "
| 1 | \n", + "dns | \n", + "openshift-dns | \n", + "oct-03-14-compute | \n", + "dns-default-6hbkp | \n", + "2022-06-08 13:31:00 | \n", + "0.004912 | \n", + "0.004482 | \n", + "0.005205 | \n", + "0.004894 | \n", + "0.004894 | \n", + "