Skip to content

Some data transformations on missing values #56

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
IreneThoma wants to merge 13 commits into
base: master
Choose a base branch
from
Open

Some data transformations on missing values #56

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
004478a
remove unnecessary code from repo
YariJulianAnrango Jun 6, 2023
e5a5aad
add apple data parser
YariJulianAnrango Jun 6, 2023
70f5b73
source from parser added
YariJulianAnrango Jun 6, 2023
93e8268
add script to combine steps with weather data
YariJulianAnrango Jun 7, 2023
037d132
add function to remove some features
IreneThoma Jun 8, 2023
d707c44
Missing data changes
IreneThoma Jun 9, 2023
b9f61e3
Add files via upload
IreneThoma Jun 10, 2023
f9829f2
Rename missing_data.ipynb to missing_data_OLD.ipynb
IreneThoma Jun 11, 2023
855d23c
Add files via upload
IreneThoma Jun 11, 2023
d981cc2
Opslaan workspace missing values
IreneThoma Jun 13, 2023
3573cfb
removed merge conflicts
IreneThoma Jun 13, 2023
2ef900b
create python files
IreneThoma Jun 14, 2023
f85db96
delete csv
IreneThoma Jun 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,49 @@
datasets
PythonCode/intermediate_datafiles/
PythonCode/Example_graphs/
weather_steps.csv
data_used/StepCount.csv
exportxml/ActiveEnergyBurned.csv
.gitignore
exportxml/ActivitySummary.csv
exportxml/AppleExerciseTime.csv
exportxml/AppleStandHour.csv
exportxml/AppleStandTime.csv
exportxml/BasalEnergyBurned.csv
exportxml/DietaryWater.csv
exportxml/DistanceCycling.csv
exportxml/DistanceWalkingRunning.csv
exportxml/EnvironmentalAudioExposure.csv
exportxml/EnvironmentalSoundReduction.csv
exportxml/FlightsClimbed.csv
exportxml/HeadphoneAudioExposure.csv
exportxml/HeartRate.csv
exportxml/HeartRateVariabilitySDNN.csv
exportxml/README.md
exportxml/RespiratoryRate.csv
exportxml/RestingHeartRate.csv
exportxml/SleepAnalysis.csv
exportxml/StairAscentSpeed.csv
exportxml/StairDescentSpeed.csv
exportxml/StepCount.csv
exportxml/WalkingAsymmetryPercentage.csv
exportxml/WalkingDoubleSupportPercentage.csv
exportxml/WalkingHeartRateAverage.csv
exportxml/WalkingSpeed.csv
exportxml/WalkingStepLength.csv
exportxml/Workout.csv
.gitignore
Missing_data/missing_data_transformation.ipynb
Missing_data/missing_data_transfromation.py
combine_data.py
export.xml
README.md
data_used/weather.txt
README.md
Missing_data/weather_steps_merged.csv
Missing_data/New_weather_steps.csv
Missing_data/transformation_notgood.py
New_weather_steps.csv
Steps_weather_combined.csv
Missing_data/EDA_old.ipynb
Transformed_weather_steps.csv
340 changes: 340 additions & 0 deletions Missing_data/apply_interpolation.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,340 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import numpy as np"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>start</th>\n",
" <th>steps</th>\n",
" <th>combined</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2023-01-01 00:00:00</td>\n",
" <td>0.000000</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2023-01-01 00:10:00</td>\n",
" <td>97.054054</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2023-01-01 00:20:00</td>\n",
" <td>94.945946</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>2023-01-01 00:30:00</td>\n",
" <td>0.000000</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2023-01-01 00:40:00</td>\n",
" <td>46.876667</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11647</th>\n",
" <td>2023-06-05 21:10:00</td>\n",
" <td>0.000000</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11648</th>\n",
" <td>2023-06-05 21:20:00</td>\n",
" <td>0.000000</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11649</th>\n",
" <td>2023-06-05 21:30:00</td>\n",
" <td>0.000000</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11650</th>\n",
" <td>2023-06-05 21:40:00</td>\n",
" <td>0.000000</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11651</th>\n",
" <td>2023-06-05 21:50:00</td>\n",
" <td>0.000000</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>11652 rows × 3 columns</p>\n",
"</div>"
],
"text/plain": [
" start steps combined\n",
"0 2023-01-01 00:00:00 0.000000 5.0\n",
"1 2023-01-01 00:10:00 97.054054 5.0\n",
"2 2023-01-01 00:20:00 94.945946 5.0\n",
"3 2023-01-01 00:30:00 0.000000 5.0\n",
"4 2023-01-01 00:40:00 46.876667 5.0\n",
"... ... ... ...\n",
"11647 2023-06-05 21:10:00 0.000000 5.0\n",
"11648 2023-06-05 21:20:00 0.000000 5.0\n",
"11649 2023-06-05 21:30:00 0.000000 5.0\n",
"11650 2023-06-05 21:40:00 0.000000 5.0\n",
"11651 2023-06-05 21:50:00 0.000000 5.0\n",
"\n",
"[11652 rows x 3 columns]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"New_weather_steps = pd.read_csv(\"Transformed_weather_steps.csv\")\n",
"New_weather_steps = New_weather_steps.drop(\"Unnamed: 0\",axis=1)\n",
"display(New_weather_steps)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>start</th>\n",
" <th>steps</th>\n",
" <th>combined</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2023-01-01 00:00:00</td>\n",
" <td>0.000000</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2023-01-01 00:10:00</td>\n",
" <td>97.054054</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2023-01-01 00:20:00</td>\n",
" <td>94.945946</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>2023-01-01 00:30:00</td>\n",
" <td>0.000000</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2023-01-01 00:40:00</td>\n",
" <td>46.876667</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>2023-01-01 00:50:00</td>\n",
" <td>126.326232</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>2023-01-01 02:00:00</td>\n",
" <td>0.000000</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>2023-01-01 02:10:00</td>\n",
" <td>117.962963</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>2023-01-01 02:20:00</td>\n",
" <td>134.037037</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>2023-01-01 02:30:00</td>\n",
" <td>113.000000</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>2023-01-01 02:40:00</td>\n",
" <td>11.586919</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>2023-01-01 02:50:00</td>\n",
" <td>109.013415</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>2023-01-01 06:00:00</td>\n",
" <td>0.000000</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>2023-01-01 06:10:00</td>\n",
" <td>0.000000</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>2023-01-01 06:20:00</td>\n",
" <td>0.000000</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" start steps combined\n",
"0 2023-01-01 00:00:00 0.000000 5.0\n",
"1 2023-01-01 00:10:00 97.054054 5.0\n",
"2 2023-01-01 00:20:00 94.945946 5.0\n",
"3 2023-01-01 00:30:00 0.000000 5.0\n",
"4 2023-01-01 00:40:00 46.876667 5.0\n",
"5 2023-01-01 00:50:00 126.326232 5.0\n",
"6 2023-01-01 02:00:00 0.000000 5.0\n",
"7 2023-01-01 02:10:00 117.962963 5.0\n",
"8 2023-01-01 02:20:00 134.037037 5.0\n",
"9 2023-01-01 02:30:00 113.000000 5.0\n",
"10 2023-01-01 02:40:00 11.586919 5.0\n",
"11 2023-01-01 02:50:00 109.013415 5.0\n",
"12 2023-01-01 06:00:00 0.000000 5.0\n",
"13 2023-01-01 06:10:00 0.000000 5.0\n",
"14 2023-01-01 06:20:00 0.000000 5.0"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"New_weather_steps['steps'] = New_weather_steps['steps'].interpolate()\n",
"display(New_weather_steps.iloc[:15])"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"start 0\n",
"steps 0\n",
"combined 0\n",
"dtype: int64\n"
]
}
],
"source": [
"print(New_weather_steps.isna().sum())"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.13"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
Loading