+
+
+
+
+
+
+
+
+Smart Parking Demand — Use Case¶
Executive Summary¶
This analysis identifies clear temporal and spatial parking demand patterns across Melbourne zones, showing that machine learning models can recover useful structure in the data, but the current baseline performance is moderate rather than deployment-ready.
+Authored by: Shrikrishna Rai
+Duration: 60 mins
+Level: Intermediate
+Pre-requisite Skills: Python, Pandas, Matplotlib, scikit-learn
+Overview¶
This analysis provides a comprehensive assessment of hourly parking demand patterns across Melbourne's on-street parking zones. By integrating sensor data with restriction metadata, we quantify occupancy trends and develop predictive models to support parking management, pricing optimization, and infrastructure planning decisions.
+Methodology¶
The analysis ingests historical parking sensor data and live occupancy records from Melbourne's open data platform, combines them with parking restriction information, and produces a normalized hourly demand dataset. Machine learning models (Logistic Regression and Random Forest) are applied to classify demand levels (Low, Medium, High) based on temporal and spatial features, with cross-validation and a temporal holdout used to estimate generalization more realistically.
+Datasets used¶
All datasets are sourced from Melbourne Open Data and provide comprehensive coverage of on-street parking activities, restrictions, and bay status:
+-
+
On-street Car Parking Sensor Data (2014) – Historical parking events collected from bay-level sensors deployed across Melbourne. Contains arrival and departure timestamps, device identifiers, duration of stay, and street information. This dataset provides granular temporal patterns of parking occupancy throughout 2014, enabling analysis of demand by hour, day, and location.
+
+Parking Bay Arrivals and Departures (2014) – Event-level records of individual parking bay occupancy transitions. Captures each arrival and departure event with precise timestamps and bay identifiers, complementing the sensor data with explicit state change information for more accurate occupancy calculations.
+
+On-Street Parking Bay Sensors (Live) – Real-time API endpoint providing current occupancy status of parking bays across Melbourne's CBD and surrounding areas. Includes bay status (Present/Absent), last update timestamp, kerbside identifiers, and location coordinates (latitude/longitude). Updated dynamically to reflect current bay availability.
+
+Sign Plates Located in Each Parking Zone – Parking restriction metadata including zone numbers, restriction types (e.g., 2P, 4P, permit-only, disabled), time restrictions, and zone coordinates. This dataset enables spatial analysis and linking restrictions to occupancy patterns.
+
+
Data Processing and Analysis Pipeline¶
The following sections execute the complete workflow: data ingestion from Melbourne Open Data sources, cleaning and standardization, feature engineering for temporal and spatial analysis, exploratory data analysis, and machine learning model development with performance validation.
+
+
+
+
+
+
+
+In [238]:
+
+
+
+
+
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+import requests
+import zipfile
+from pathlib import Path
+
+from sklearn.compose import ColumnTransformer
+from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
+from sklearn.impute import SimpleImputer
+from sklearn.linear_model import LogisticRegression
+from sklearn.metrics import (
+ accuracy_score,
+ balanced_accuracy_score,
+ classification_report,
+ confusion_matrix,
+ ConfusionMatrixDisplay,
+ f1_score,
+)
+from sklearn.model_selection import StratifiedKFold, cross_validate, train_test_split
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import OneHotEncoder, StandardScaler
+
+
+
+
+
+
+
+
+In [239]:
+
+
+
+
+
+# Local path where raw downloads are cached (ensure directories exist)
+RAW_DATA_DIR = Path("../data/raw")
+RAW_DATA_DIR.mkdir(parents=True, exist_ok=True)
+
+# Cache directory for fetched JSON API responses
+CACHE_DIR = Path("../data/cache")
+CACHE_DIR.mkdir(parents=True, exist_ok=True)
+
+# Directory for processed outputs
+PROCESSED_DIR = Path("../data/processed")
+PROCESSED_DIR.mkdir(parents=True, exist_ok=True)
+
+# Remote datasets (zips) used in this notebook
+CAR_PARKING_DATA_ZIP_URL = "https://opendatasoft-s3.s3.amazonaws.com/downloads/archive/t6hb-9uf2.zip"
+PARKING_BAY_DATA_ZIP_URL = "https://opendatasoft-s3.s3.amazonaws.com/downloads/archive/mq3i-cbxd.zip"
+
+
+def load_melbourne_api(dataset_slug, rows=5000, use_cache=True):
+ """Load a dataset from the Melbourne open data API and return a DataFrame.
+
+ Optional JSON caching is available under `../data/cache` to avoid
+ re-fetching during development runs.
+ """
+ url = "https://data.melbourne.vic.gov.au/api/records/1.0/search/"
+ params = {"dataset": dataset_slug, "rows": rows}
+
+ cache_file = CACHE_DIR / f"{dataset_slug}.json"
+ if use_cache and cache_file.exists():
+ import json
+ with cache_file.open("r", encoding="utf-8") as fh:
+ data = json.load(fh)
+ else:
+ response = requests.get(url, params=params, timeout=60)
+ response.raise_for_status()
+ data = response.json()
+ if use_cache:
+ import json
+ with cache_file.open("w", encoding="utf-8") as fh:
+ json.dump(data, fh)
+
+ records = data.get("records", [])
+ return pd.DataFrame([record["fields"] for record in records])
+
+
+def download_file(url, destination):
+ """Download a file (streamed) and save to the destination path.
+
+ The function creates parent directories as needed and writes in chunks
+ to avoid excessive memory usage for large files.
+ """
+ destination = Path(destination)
+ destination.parent.mkdir(parents=True, exist_ok=True)
+
+ response = requests.get(url, stream=True, timeout=120)
+ response.raise_for_status()
+
+ with destination.open("wb") as output_file:
+ for chunk in response.iter_content(chunk_size=1024 * 1024):
+ if chunk:
+ output_file.write(chunk)
+
+ return destination
+
+
+def read_cached_zip_csv(zip_url, cache_name, nrows=None):
+ """Download (if needed) a zip file and return the first CSV inside as a DataFrame.
+
+ Args:
+ zip_url: remote URL pointing to the zip archive
+ cache_name: local filename to use for caching the downloaded zip
+ nrows: optional limit for number of CSV rows to read (useful for development)
+ """
+ cache_path = RAW_DATA_DIR / cache_name
+
+ if not cache_path.exists():
+ # Download once and reuse the local cached file
+ download_file(zip_url, cache_path)
+
+ with zipfile.ZipFile(cache_path) as archive:
+ csv_member = next(
+ (name for name in archive.namelist() if name.lower().endswith(".csv")),
+ None,
+ )
+
+ if csv_member is None:
+ raise ValueError(f"No CSV file found inside {cache_path.name}")
+
+ with archive.open(csv_member) as csv_file:
+ return pd.read_csv(csv_file, nrows=nrows)
+
+
+
+
+
+
+
+
+
+Data Ingestion¶
This section downloads the Melbourne open data sources when they are not already cached locally, then loads each dataset into a pandas DataFrame. The row limits keep the notebook responsive while still providing enough observations for analysis.
+
+
+
+
+
+
+
+
+In [240]:
+
+
+
+
+
+# --- Load datasets (may take time on first run) ---
+on_street_parking_bay_sensor_df = load_melbourne_api("on-street-parking-bay-sensors", rows=5000)
+
+sign_plates_located_in_parking_bays_df = load_melbourne_api(
+ "sign-plates-located-in-each-parking-zone",
+ rows=5000,
+)
+
+on_street_car_parking_sensor_df = read_cached_zip_csv(
+ CAR_PARKING_DATA_ZIP_URL,
+ "On-street_Car_Parking_Sensor_Data_-_2014.zip",
+ nrows=300000,
+)
+
+parking_bay_arrivals_and_departures_df = read_cached_zip_csv(
+ PARKING_BAY_DATA_ZIP_URL,
+ "Parking_bay_arrivals_and_departures_2014.zip",
+ nrows=300000,
+)
+
+
+
+
+
+
+
+
+
+
+
+/tmp/ipykernel_10868/2911588302.py:89: DtypeWarning: Columns (0: StreetId, 1: BetweenStreet1 Id, 2: BetweenStreet2 Id) have mixed types. Specify dtype option on import or set low_memory=False. + return pd.read_csv(csv_file, nrows=nrows) ++
+
+
+
+
+
+
+
+
+Initial Dataset Checks¶
Preview each raw dataset, inspect the column names, and check the inferred data types before cleaning. These checks make schema differences visible early.
+
+
+
+
+
+
+
+In [241]:
+
+
+
+
+
+print("**on_street_car_parking_sensor_df**")
+print(on_street_car_parking_sensor_df.head())
+print(on_street_car_parking_sensor_df.columns)
+on_street_car_parking_sensor_df.info()
+
+
+
+
+
+
+
+
+
+
+
+**on_street_car_parking_sensor_df** + DeviceId ArrivalTime DepartureTime DurationSeconds StreetMarker Sign \ +0 10,032 07/01/2014 11:43:23 AM 07/01/2014 11:43:30 AM 7 3273S 1P MTR M-F 9:30-19:30 +1 11,649 03/19/2014 10:09:38 AM 03/19/2014 10:11:22 AM 104 2206N 1/2P MTR M-SAT 7:30-19:30 +2 9,126 06/12/2014 07:30:00 AM 06/12/2014 10:17:23 AM 10,043 11921W 2P MTR M-F 7:30-18:30 +3 12,462 07/02/2014 07:30:11 AM 07/02/2014 07:34:11 AM 240 2418N 1/4P M-SAT 7:30-9:30 +4 14,588 07/16/2014 08:12:00 AM 07/16/2014 08:12:14 AM 14 13749W 2P TKT A M-SAT 7:30-18:30 + + Area StreetId StreetName BetweenStreet1 BetweenStreet2 Side Of Street In Violation \ +0 County 894 LONSDALE STREET KING STREET WILLIAM STREET 4.0 False +1 McKillop 911 Lt COLLINS STREET QUEEN STREET ELIZABETH STREET 3.0 False +2 East Melbourne 511 CLARENDON STREET GEORGE STREET WELLINGTON PARADE 5.0 False +3 RACV 123 BOURKE STREET WILLIAM STREET QUEEN STREET 3.0 False +4 Docklands 1,398 WATERFRONT WAY DOCKLANDS DRIVE Lt DOCKLANDS DRIVE 5.0 False + + Vehicle Present +0 False +1 False +2 False +3 True +4 False +Index(['DeviceId', 'ArrivalTime', 'DepartureTime', 'DurationSeconds', 'StreetMarker', 'Sign', 'Area', 'StreetId', + 'StreetName', 'BetweenStreet1', 'BetweenStreet2', 'Side Of Street', 'In Violation', 'Vehicle Present'], + dtype='str') +<class 'pandas.DataFrame'> +RangeIndex: 300000 entries, 0 to 299999 +Data columns (total 14 columns): + # Column Non-Null Count Dtype +--- ------ -------------- ----- + 0 DeviceId 300000 non-null str + 1 ArrivalTime 300000 non-null str + 2 DepartureTime 300000 non-null str + 3 DurationSeconds 300000 non-null str + 4 StreetMarker 299981 non-null str + 5 Sign 288775 non-null str + 6 Area 299981 non-null str + 7 StreetId 299981 non-null str + 8 StreetName 299981 non-null str + 9 BetweenStreet1 299981 non-null str + 10 BetweenStreet2 299697 non-null str + 11 Side Of Street 299981 non-null float64 + 12 In Violation 300000 non-null bool + 13 Vehicle Present 300000 non-null bool +dtypes: bool(2), float64(1), str(11) +memory usage: 28.0 MB ++
+
+
+
+
+
+
+In [242]:
+
+
+
+
+
+print("**on_street_parking_bay_sensor_df**")
+print(on_street_parking_bay_sensor_df.head())
+print(on_street_parking_bay_sensor_df.columns)
+on_street_parking_bay_sensor_df.info()
+
+
+
+
+
+
+
+
+
+
+
+**on_street_parking_bay_sensor_df** + status_timestamp zone_number lastupdated kerbsideid status_description \ +0 2024-08-18T08:23:46+00:00 7394.0 2024-12-30T00:44:37+00:00 9344 Unoccupied +1 2024-08-14T09:22:05+00:00 7392.0 2025-01-10T02:44:36+00:00 9373 Unoccupied +2 2024-11-28T02:30:57+00:00 7084.0 2024-12-04T23:44:37+00:00 8735 Present +3 2024-11-27T23:23:40+00:00 7084.0 2024-12-04T23:44:37+00:00 8749 Unoccupied +4 2023-08-21T00:09:03+00:00 7800.0 2025-01-09T05:44:36+00:00 24505 Present + + location +0 [-37.80494402936792, 144.95916129121264] +1 [-37.80329074425858, 144.95836336946644] +2 [-37.80223335664963, 144.96120480793184] +3 [-37.80230361629087, 144.9618505999636] +4 [-37.79756250415984, 144.95759881813447] +Index(['status_timestamp', 'zone_number', 'lastupdated', 'kerbsideid', 'status_description', 'location'], dtype='str') +<class 'pandas.DataFrame'> +RangeIndex: 3309 entries, 0 to 3308 +Data columns (total 6 columns): + # Column Non-Null Count Dtype +--- ------ -------------- ----- + 0 status_timestamp 3309 non-null str + 1 zone_number 3080 non-null float64 + 2 lastupdated 3309 non-null str + 3 kerbsideid 3309 non-null int64 + 4 status_description 3309 non-null str + 5 location 3309 non-null object +dtypes: float64(1), int64(1), object(1), str(3) +memory usage: 155.2+ KB ++
+
+
+
+
+
+
+In [243]:
+
+
+
+
+
+print("**parking_bay_arrivals_and_departures_df**")
+print(parking_bay_arrivals_and_departures_df.head())
+print(parking_bay_arrivals_and_departures_df.columns)
+parking_bay_arrivals_and_departures_df.info()
+
+
+
+
+
+
+
+
+
+
+
+**parking_bay_arrivals_and_departures_df** + ParkingEventId DeviceId ArrivalTime DepartureTime DurationSeconds StreetMarker \ +0 3,123 11,710 04/02/2014 10:56:29 AM +0000 04/02/2014 11:06:08 AM +0000 600 6588N +1 3,124 11,710 04/02/2014 11:07:23 AM +0000 04/02/2014 11:38:48 AM +0000 1,860 6588N +2 3,125 11,710 04/02/2014 12:09:59 PM +0000 04/02/2014 12:34:30 PM +0000 1,500 6588N +3 3,126 11,710 04/02/2014 12:39:30 PM +0000 04/02/2014 12:58:07 PM +0000 1,140 6588N +4 3,129 11,710 04/02/2014 01:08:46 PM +0000 04/02/2014 01:14:46 PM +0000 360 6588N + + SignPlateId Sign Area AreaName StreetId StreetName BetweenStreet1 Id \ +0 6 1P MTR M-SAT 7:30-19:30 12 Victoria Market 681 FRANKLIN STREET 1,171 +1 6 1P MTR M-SAT 7:30-19:30 12 Victoria Market 681 FRANKLIN STREET 1,171 +2 6 1P MTR M-SAT 7:30-19:30 12 Victoria Market 681 FRANKLIN STREET 1,171 +3 6 1P MTR M-SAT 7:30-19:30 12 Victoria Market 681 FRANKLIN STREET 1,171 +4 6 1P MTR M-SAT 7:30-19:30 12 Victoria Market 681 FRANKLIN STREET 1,171 + + BetweenStreet1 Description BetweenStreet2 Id BetweenStreet2 Description SideOfStreet SideCode SideName BayID \ +0 QUEEN STREET 627 ELIZABETH STREET 3 N North 3,495 +1 QUEEN STREET 627 ELIZABETH STREET 3 N North 3,495 +2 QUEEN STREET 627 ELIZABETH STREET 3 N North 3,495 +3 QUEEN STREET 627 ELIZABETH STREET 3 N North 3,495 +4 QUEEN STREET 627 ELIZABETH STREET 3 N North 3,495 + + InViolation +0 False +1 False +2 False +3 False +4 False +Index(['ParkingEventId', 'DeviceId', 'ArrivalTime', 'DepartureTime', 'DurationSeconds', 'StreetMarker', 'SignPlateId', + 'Sign', 'Area', 'AreaName', 'StreetId', 'StreetName', 'BetweenStreet1 Id', 'BetweenStreet1 Description', + 'BetweenStreet2 Id', 'BetweenStreet2 Description', 'SideOfStreet', 'SideCode', 'SideName', 'BayID', + 'InViolation'], + dtype='str') +<class 'pandas.DataFrame'> +RangeIndex: 300000 entries, 0 to 299999 +Data columns (total 21 columns): + # Column Non-Null Count Dtype +--- ------ -------------- ----- + 0 ParkingEventId 300000 non-null str + 1 DeviceId 300000 non-null str + 2 ArrivalTime 300000 non-null str + 3 DepartureTime 300000 non-null str + 4 DurationSeconds 300000 non-null str + 5 StreetMarker 300000 non-null str + 6 SignPlateId 300000 non-null int64 + 7 Sign 281953 non-null str + 8 Area 300000 non-null int64 + 9 AreaName 300000 non-null str + 10 StreetId 300000 non-null object + 11 StreetName 300000 non-null str + 12 BetweenStreet1 Id 300000 non-null object + 13 BetweenStreet1 Description 300000 non-null str + 14 BetweenStreet2 Id 300000 non-null object + 15 BetweenStreet2 Description 300000 non-null str + 16 SideOfStreet 300000 non-null int64 + 17 SideCode 300000 non-null str + 18 SideName 300000 non-null str + 19 BayID 300000 non-null str + 20 InViolation 300000 non-null bool +dtypes: bool(1), int64(3), object(3), str(14) +memory usage: 46.1+ MB ++
+
+
+
+
+
+
+
+In [244]:
+
+
+
+
+
+print("**sign_plates_located_in_parking_bays_df**")
+print(sign_plates_located_in_parking_bays_df.head())
+print(sign_plates_located_in_parking_bays_df.columns)
+sign_plates_located_in_parking_bays_df.info()
+
+
+
+
+
+
+
+
+
+
+
+**sign_plates_located_in_parking_bays_df** + restriction_display parkingzone time_restrictions_start restriction_days time_restrictions_finish +0 2P 7001 07:30:00 Sat 12:30:00 +1 2P 7007 07:30:00 Sat 12:30:00 +2 2P 7025 07:30:00 Mon-Fri 18:30:00 +3 2P 7025 07:30:00 Sat 12:30:00 +4 HP 7019 07:30:00 Sat 12:30:00 +Index(['restriction_display', 'parkingzone', 'time_restrictions_start', 'restriction_days', + 'time_restrictions_finish'], + dtype='str') +<class 'pandas.DataFrame'> +RangeIndex: 2033 entries, 0 to 2032 +Data columns (total 5 columns): + # Column Non-Null Count Dtype +--- ------ -------------- ----- + 0 restriction_display 2033 non-null str + 1 parkingzone 2033 non-null int64 + 2 time_restrictions_start 2033 non-null str + 3 restriction_days 2033 non-null str + 4 time_restrictions_finish 2033 non-null str +dtypes: int64(1), str(4) +memory usage: 79.5 KB ++
+
+
+
+
+
+
+
+
+Data Cleaning and Preparation¶
The raw sources use different naming conventions and data formats. The next cells standardise column names, remove duplicate rows, convert numeric and datetime fields, and drop records that are missing fields required for demand analysis.
+
+
+
+
+
+
+
+In [245]:
+
+
+
+
+
+def clean_column_names(df):
+ df = df.copy()
+ df.columns = (
+ df.columns.str.strip()
+ .str.lower()
+ .str.replace(" ", "_")
+ .str.replace("-", "_")
+ )
+ return df
+
+on_street_car_parking_sensor_df = clean_column_names(on_street_car_parking_sensor_df)
+on_street_parking_bay_sensor_df = clean_column_names(on_street_parking_bay_sensor_df)
+parking_bay_arrivals_and_departures_df = clean_column_names(parking_bay_arrivals_and_departures_df)
+sign_plates_located_in_parking_bays_df = clean_column_names(sign_plates_located_in_parking_bays_df)
+
+
+
+
+
+
+
+In [246]:
+
+
+
+
+
+print("on_street_car_parking_sensor_df columns:")
+print(on_street_car_parking_sensor_df.columns.tolist())
+
+print("\non_street_parking_bay_sensor_df columns:")
+print(on_street_parking_bay_sensor_df.columns.tolist())
+
+print("\nparking_bay_arrivals_and_departures_df columns:")
+print(parking_bay_arrivals_and_departures_df.columns.tolist())
+
+print("\nsign_plates_located_in_parking_bays_df columns:")
+print(sign_plates_located_in_parking_bays_df.columns.tolist())
+
+
+
+
+
+
+
+
+
+
+
+on_street_car_parking_sensor_df columns: +['deviceid', 'arrivaltime', 'departuretime', 'durationseconds', 'streetmarker', 'sign', 'area', 'streetid', 'streetname', 'betweenstreet1', 'betweenstreet2', 'side_of_street', 'in_violation', 'vehicle_present'] + +on_street_parking_bay_sensor_df columns: +['status_timestamp', 'zone_number', 'lastupdated', 'kerbsideid', 'status_description', 'location'] + +parking_bay_arrivals_and_departures_df columns: +['parkingeventid', 'deviceid', 'arrivaltime', 'departuretime', 'durationseconds', 'streetmarker', 'signplateid', 'sign', 'area', 'areaname', 'streetid', 'streetname', 'betweenstreet1_id', 'betweenstreet1_description', 'betweenstreet2_id', 'betweenstreet2_description', 'sideofstreet', 'sidecode', 'sidename', 'bayid', 'inviolation'] + +sign_plates_located_in_parking_bays_df columns: +['restriction_display', 'parkingzone', 'time_restrictions_start', 'restriction_days', 'time_restrictions_finish'] ++
+
+
+
+
+
+
+
+In [247]:
+
+
+
+
+
+on_street_car_parking_sensor_df = on_street_car_parking_sensor_df.drop_duplicates()
+on_street_parking_bay_sensor_df["location"] = (
+ on_street_parking_bay_sensor_df["location"].apply(
+ lambda value: tuple(value) if isinstance(value, list) else value
+ )
+)
+on_street_parking_bay_sensor_df = on_street_parking_bay_sensor_df.drop_duplicates()
+parking_bay_arrivals_and_departures_df = parking_bay_arrivals_and_departures_df.drop_duplicates()
+sign_plates_located_in_parking_bays_df = sign_plates_located_in_parking_bays_df.drop_duplicates()
+
+
+
+
+
+
+
+
+
+Numeric and Datetime Normalisation¶
Some downloaded CSV fields include comma-formatted numbers or date strings. These cells convert them into numeric and datetime types so pandas can group, filter, and model them correctly.
+
+
+
+
+
+
+
+In [248]:
+
+
+
+
+
+# Remove comma separators from 2014 sensor identifiers and durations
+on_street_car_parking_sensor_df["deviceid"] = (
+ on_street_car_parking_sensor_df["deviceid"].astype(str).str.replace(",", "", regex=False)
+)
+on_street_car_parking_sensor_df["streetid"] = (
+ on_street_car_parking_sensor_df["streetid"].astype(str).str.replace(",", "", regex=False)
+)
+on_street_car_parking_sensor_df["durationseconds"] = (
+ on_street_car_parking_sensor_df["durationseconds"].astype(str).str.replace(",", "", regex=False)
+)
+
+# Remove comma separators from arrivals/departures identifiers and durations
+parking_bay_arrivals_and_departures_df["parkingeventid"] = (
+ parking_bay_arrivals_and_departures_df["parkingeventid"].astype(str).str.replace(",", "", regex=False)
+)
+parking_bay_arrivals_and_departures_df["deviceid"] = (
+ parking_bay_arrivals_and_departures_df["deviceid"].astype(str).str.replace(",", "", regex=False)
+)
+parking_bay_arrivals_and_departures_df["durationseconds"] = (
+ parking_bay_arrivals_and_departures_df["durationseconds"].astype(str).str.replace(",", "", regex=False)
+)
+parking_bay_arrivals_and_departures_df["bayid"] = (
+ parking_bay_arrivals_and_departures_df["bayid"].astype(str).str.replace(",", "", regex=False)
+)
+
+
+
+
+
+
+
+In [249]:
+
+
+
+
+
+on_street_car_parking_sensor_df["deviceid"] = pd.to_numeric(on_street_car_parking_sensor_df["deviceid"], errors="coerce")
+on_street_car_parking_sensor_df["streetid"] = pd.to_numeric(on_street_car_parking_sensor_df["streetid"], errors="coerce")
+on_street_car_parking_sensor_df["durationseconds"] = pd.to_numeric(on_street_car_parking_sensor_df["durationseconds"], errors="coerce")
+
+parking_bay_arrivals_and_departures_df["parkingeventid"] = pd.to_numeric(parking_bay_arrivals_and_departures_df["parkingeventid"], errors="coerce")
+parking_bay_arrivals_and_departures_df["deviceid"] = pd.to_numeric(parking_bay_arrivals_and_departures_df["deviceid"], errors="coerce")
+parking_bay_arrivals_and_departures_df["durationseconds"] = pd.to_numeric(parking_bay_arrivals_and_departures_df["durationseconds"], errors="coerce")
+parking_bay_arrivals_and_departures_df["bayid"] = pd.to_numeric(parking_bay_arrivals_and_departures_df["bayid"], errors="coerce")
+
+
+
+
+
+
+
+In [250]:
+
+
+
+
+
+on_street_car_parking_sensor_df["arrivaltime"] = pd.to_datetime(
+ on_street_car_parking_sensor_df["arrivaltime"],
+ errors="coerce"
+)
+
+on_street_car_parking_sensor_df["departuretime"] = pd.to_datetime(
+ on_street_car_parking_sensor_df["departuretime"],
+ errors="coerce"
+)
+
+on_street_parking_bay_sensor_df["lastupdated"] = pd.to_datetime(
+ on_street_parking_bay_sensor_df["lastupdated"],
+ errors="coerce",
+ utc=True
+)
+
+on_street_parking_bay_sensor_df["status_timestamp"] = pd.to_datetime(
+ on_street_parking_bay_sensor_df["status_timestamp"],
+ errors="coerce",
+ utc=True
+)
+
+parking_bay_arrivals_and_departures_df["arrivaltime"] = pd.to_datetime(
+ parking_bay_arrivals_and_departures_df["arrivaltime"],
+ errors="coerce"
+)
+
+parking_bay_arrivals_and_departures_df["departuretime"] = pd.to_datetime(
+ parking_bay_arrivals_and_departures_df["departuretime"],
+ errors="coerce"
+)
+
+
+
+
+
+
+
+
+In [251]:
+
+
+
+
+
+on_street_car_parking_sensor_df = on_street_car_parking_sensor_df.dropna(
+ subset=["deviceid", "arrivaltime", "departuretime"]
+)
+
+on_street_parking_bay_sensor_df = on_street_parking_bay_sensor_df.dropna(
+ subset=["status_timestamp", "status_description", "kerbsideid"]
+)
+
+parking_bay_arrivals_and_departures_df = parking_bay_arrivals_and_departures_df.dropna(
+ subset=["deviceid", "arrivaltime", "departuretime", "bayid"]
+)
+
+sign_plates_located_in_parking_bays_df = sign_plates_located_in_parking_bays_df.dropna(
+ subset=["parkingzone"]
+)
+
+
+
+
+
+
+
+
+
+Temporal Feature Engineering¶
Arrival and status timestamps are transformed into hour, day, month, and weekend indicators. These features capture the recurring time patterns that usually drive parking demand.
+
+
+
+
+
+
+
+In [252]:
+
+
+
+
+
+# Add temporal features from historical sensor arrivals
+on_street_car_parking_sensor_df["arrival_hour"] = on_street_car_parking_sensor_df["arrivaltime"].dt.hour
+on_street_car_parking_sensor_df["arrival_day"] = on_street_car_parking_sensor_df["arrivaltime"].dt.day_name()
+on_street_car_parking_sensor_df["arrival_month"] = on_street_car_parking_sensor_df["arrivaltime"].dt.month
+on_street_car_parking_sensor_df["is_weekend"] = on_street_car_parking_sensor_df["arrivaltime"].dt.dayofweek >= 5
+
+# Add temporal features from live bay status timestamps
+on_street_parking_bay_sensor_df["status_hour"] = on_street_parking_bay_sensor_df["status_timestamp"].dt.hour
+on_street_parking_bay_sensor_df["status_day"] = on_street_parking_bay_sensor_df["status_timestamp"].dt.day_name()
+on_street_parking_bay_sensor_df["status_month"] = on_street_parking_bay_sensor_df["status_timestamp"].dt.month
+on_street_parking_bay_sensor_df["is_weekend"] = on_street_parking_bay_sensor_df["status_timestamp"].dt.dayofweek >= 5
+
+# Add temporal features from arrivals/departures timestamps
+parking_bay_arrivals_and_departures_df["arrival_hour"] = parking_bay_arrivals_and_departures_df["arrivaltime"].dt.hour
+parking_bay_arrivals_and_departures_df["arrival_day"] = parking_bay_arrivals_and_departures_df["arrivaltime"].dt.day_name()
+parking_bay_arrivals_and_departures_df["arrival_month"] = parking_bay_arrivals_and_departures_df["arrivaltime"].dt.month
+parking_bay_arrivals_and_departures_df["is_weekend"] = parking_bay_arrivals_and_departures_df["arrivaltime"].dt.dayofweek >= 5
+
+
+
+
+
+
+
+
+In [253]:
+
+
+
+
+
+# Keep timestamps timezone-naive so they are easier to read and export
+on_street_parking_bay_sensor_df["lastupdated"] = (
+ on_street_parking_bay_sensor_df["lastupdated"].dt.tz_convert(None)
+)
+
+on_street_parking_bay_sensor_df["status_timestamp"] = (
+ on_street_parking_bay_sensor_df["status_timestamp"].dt.tz_convert(None)
+)
+
+
+
+
+
+
+
+
+
+Occupancy and Zone Metadata¶
Live bay status is converted into a binary occupancy flag, then joined with parking restriction metadata by zone. This creates the main analysis table used to calculate demand.
+
+
+
+
+
+
+
+In [254]:
+
+
+
+
+
+on_street_parking_bay_sensor_df["occupied"] = on_street_parking_bay_sensor_df["status_description"].apply(
+ lambda x: 1 if x == "Present" else 0
+)
+
+on_street_parking_bay_sensor_df["occupied"].value_counts()
+
+
+
+
+
+
+
+
+Out[254]:
+
+
+occupied +1 1885 +0 1424 +Name: count, dtype: int64+
+
+
+
+
+
+
+In [255]:
+
+
+
+
+
+sign_plates_located_in_parking_bays_df = sign_plates_located_in_parking_bays_df.rename(
+ columns={"parkingzone": "zone_number"}
+)
+
+sign_plates_located_in_parking_bays_df.head()
+
+
+
+
+
+
+
+
+Out[255]:
+
+
+
+
+
+
+
+
+| + | restriction_display | +zone_number | +time_restrictions_start | +restriction_days | +time_restrictions_finish | +
|---|---|---|---|---|---|
| 0 | +2P | +7001 | +07:30:00 | +Sat | +12:30:00 | +
| 1 | +2P | +7007 | +07:30:00 | +Sat | +12:30:00 | +
| 2 | +2P | +7025 | +07:30:00 | +Mon-Fri | +18:30:00 | +
| 3 | +2P | +7025 | +07:30:00 | +Sat | +12:30:00 | +
| 4 | +HP | +7019 | +07:30:00 | +Sat | +12:30:00 | +
+
+
+
+
+
+
+In [256]:
+
+
+
+
+
+merged_bay_df = on_street_parking_bay_sensor_df.merge(
+ sign_plates_located_in_parking_bays_df,
+ on="zone_number",
+ how="left"
+)
+
+print("Merged shape:", merged_bay_df.shape)
+display(merged_bay_df.head())
+
+
+
+
+
+
+
+
+
+
+
+Merged shape: (7996, 15) ++
+
+
+
+
+
+
+
+
+
+
+| + | status_timestamp | +zone_number | +lastupdated | +kerbsideid | +status_description | +location | +status_hour | +status_day | +status_month | +is_weekend | +occupied | +restriction_display | +time_restrictions_start | +restriction_days | +time_restrictions_finish | +
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | +2024-08-18 08:23:46 | +7394.0 | +2024-12-30 00:44:37 | +9344 | +Unoccupied | +(-37.80494402936792, 144.95916129121264) | +8 | +Sunday | +8 | +True | +0 | +1P | +07:30:00 | +Mon-Sat | +18:30:00 | +
| 1 | +2024-08-14 09:22:05 | +7392.0 | +2025-01-10 02:44:36 | +9373 | +Unoccupied | +(-37.80329074425858, 144.95836336946644) | +9 | +Wednesday | +8 | +False | +0 | +2P | +07:30:00 | +Mon-Fri | +18:30:00 | +
| 2 | +2024-08-14 09:22:05 | +7392.0 | +2025-01-10 02:44:36 | +9373 | +Unoccupied | +(-37.80329074425858, 144.95836336946644) | +9 | +Wednesday | +8 | +False | +0 | +2P | +07:30:00 | +Sat | +12:30:00 | +
| 3 | +2024-11-28 02:30:57 | +7084.0 | +2024-12-04 23:44:37 | +8735 | +Present | +(-37.80223335664963, 144.96120480793184) | +2 | +Thursday | +11 | +False | +1 | +2P | +07:30:00 | +Sat | +12:30:00 | +
| 4 | +2024-11-28 02:30:57 | +7084.0 | +2024-12-04 23:44:37 | +8735 | +Present | +(-37.80223335664963, 144.96120480793184) | +2 | +Thursday | +11 | +False | +1 | +2P | +07:30:00 | +Mon-Fri | +18:30:00 | +
+
+
+
+
+
+
+In [257]:
+
+
+
+
+
+merged_bay_df.info()
+print("\nMissing values after merge:")
+print(merged_bay_df.isnull().sum())
+
+
+
+
+
+
+
+
+
+
+
+<class 'pandas.DataFrame'> +RangeIndex: 7996 entries, 0 to 7995 +Data columns (total 15 columns): + # Column Non-Null Count Dtype +--- ------ -------------- ----- + 0 status_timestamp 7996 non-null datetime64[us] + 1 zone_number 7767 non-null float64 + 2 lastupdated 7996 non-null datetime64[us] + 3 kerbsideid 7996 non-null int64 + 4 status_description 7996 non-null str + 5 location 7996 non-null object + 6 status_hour 7996 non-null int32 + 7 status_day 7996 non-null str + 8 status_month 7996 non-null int32 + 9 is_weekend 7996 non-null bool + 10 occupied 7996 non-null int64 + 11 restriction_display 7722 non-null str + 12 time_restrictions_start 7722 non-null str + 13 restriction_days 7722 non-null str + 14 time_restrictions_finish 7722 non-null str +dtypes: bool(1), datetime64[us](2), float64(1), int32(2), int64(2), object(1), str(6) +memory usage: 820.0+ KB + +Missing values after merge: +status_timestamp 0 +zone_number 229 +lastupdated 0 +kerbsideid 0 +status_description 0 +location 0 +status_hour 0 +status_day 0 +status_month 0 +is_weekend 0 +occupied 0 +restriction_display 274 +time_restrictions_start 274 +restriction_days 274 +time_restrictions_finish 274 +dtype: int64 ++
+
+
+
+
+
+
+
+In [258]:
+
+
+
+
+
+# We need zone_number for grouping, so remove rows where it is missing
+merged_bay_df = merged_bay_df.dropna(subset=["zone_number"])
+
+# Fill restriction details so downstream analysis has readable values
+merged_bay_df["restriction_days"] = merged_bay_df["restriction_days"].fillna("Unknown")
+merged_bay_df["restriction_display"] = merged_bay_df["restriction_display"].fillna("Unknown")
+merged_bay_df["time_restrictions_start"] = merged_bay_df["time_restrictions_start"].fillna("Unknown")
+merged_bay_df["time_restrictions_finish"] = merged_bay_df["time_restrictions_finish"].fillna("Unknown")
+
+
+
+
+
+
+
+
+
+
+The merged dataset now keeps only records with a valid zone_number, because the demand calculation groups by parking zone. Missing restriction fields are filled with Unknown so the downstream EDA and model features remain readable instead of carrying blank values.
+
+
+
+
+
+
+
+
+Demand Dataset Creation¶
Demand is defined as average occupancy for each zone, day, hour, and weekend combination. The continuous occupancy value is also bucketed into low, medium, and high demand classes for classification.
+
+
+
+
+
+
+
+In [259]:
+
+
+
+
+
+# Group occupancy by zone, day, hour, and weekend status
+demand_df = (
+ merged_bay_df.groupby(
+ ["zone_number", "status_day", "status_hour", "is_weekend"],
+ as_index=False
+ )["occupied"].mean()
+ )
+
+demand_df = demand_df.rename(columns={"occupied": "average_occupancy"})
+
+day_order = ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"]
+day_to_index = {day: idx for idx, day in enumerate(day_order)}
+demand_df["dayofweek_num"] = demand_df["status_day"].map(day_to_index)
+demand_df["hour_sin"] = np.sin(2 * np.pi * demand_df["status_hour"] / 24)
+demand_df["hour_cos"] = np.cos(2 * np.pi * demand_df["status_hour"] / 24)
+
+zone_restrictions = (
+ merged_bay_df.groupby("zone_number", as_index=False)
+ .agg(restriction_display=("restriction_display", "first"))
+ )
+demand_df = demand_df.merge(zone_restrictions, on="zone_number", how="left")
+
+# Richer zone features: occupancy variance and statistics
+zone_features = merged_bay_df.groupby("zone_number").agg({
+ "occupied": ["mean", "std"]
+}).reset_index()
+zone_features.columns = ["zone_number", "zone_occ_mean", "zone_occ_std"]
+zone_features["zone_occ_std"] = zone_features["zone_occ_std"].fillna(0)
+zone_features["zone_volatility"] = zone_features["zone_occ_std"] / (zone_features["zone_occ_mean"] + 0.01)
+demand_df = demand_df.merge(zone_features, on="zone_number", how="left")
+
+display(demand_df.head())
+print("Demand dataset shape:", demand_df.shape)
+print("New features added: zone_occ_mean, zone_occ_std, zone_volatility")
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+| + | zone_number | +status_day | +status_hour | +is_weekend | +average_occupancy | +dayofweek_num | +hour_sin | +hour_cos | +restriction_display | +zone_occ_mean | +zone_occ_std | +zone_volatility | +
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | +7010.0 | +Friday | +1 | +False | +0.0 | +4 | +0.258819 | +0.965926 | +2P | +0.7 | +0.470162 | +0.6622 | +
| 1 | +7010.0 | +Thursday | +21 | +False | +1.0 | +3 | +-0.707107 | +0.707107 | +2P | +0.7 | +0.470162 | +0.6622 | +
| 2 | +7010.0 | +Thursday | +22 | +False | +1.0 | +3 | +-0.500000 | +0.866025 | +2P | +0.7 | +0.470162 | +0.6622 | +
| 3 | +7010.0 | +Thursday | +23 | +False | +1.0 | +3 | +-0.258819 | +0.965926 | +2P | +0.7 | +0.470162 | +0.6622 | +
| 4 | +7010.0 | +Wednesday | +9 | +False | +0.0 | +2 | +0.707107 | +-0.707107 | +2P | +0.7 | +0.470162 | +0.6622 | +
+
+
+
+
+Demand dataset shape: (1573, 12) +New features added: zone_occ_mean, zone_occ_std, zone_volatility ++
+
+
+
+
+
+
+In [260]:
+
+
+
+
+
+def demand_level(value):
+ if value < 0.33:
+ return "Low"
+ elif value < 0.66:
+ return "Medium"
+ else:
+ return "High"
+
+demand_df["demand_level"] = demand_df["average_occupancy"].apply(demand_level)
+
+demand_df.head()
+
+
+
+
+
+
+
+
+Out[260]:
+
+
+
+
+
+
+
+
+| + | zone_number | +status_day | +status_hour | +is_weekend | +average_occupancy | +dayofweek_num | +hour_sin | +hour_cos | +restriction_display | +zone_occ_mean | +zone_occ_std | +zone_volatility | +demand_level | +
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | +7010.0 | +Friday | +1 | +False | +0.0 | +4 | +0.258819 | +0.965926 | +2P | +0.7 | +0.470162 | +0.6622 | +Low | +
| 1 | +7010.0 | +Thursday | +21 | +False | +1.0 | +3 | +-0.707107 | +0.707107 | +2P | +0.7 | +0.470162 | +0.6622 | +High | +
| 2 | +7010.0 | +Thursday | +22 | +False | +1.0 | +3 | +-0.500000 | +0.866025 | +2P | +0.7 | +0.470162 | +0.6622 | +High | +
| 3 | +7010.0 | +Thursday | +23 | +False | +1.0 | +3 | +-0.258819 | +0.965926 | +2P | +0.7 | +0.470162 | +0.6622 | +High | +
| 4 | +7010.0 | +Wednesday | +9 | +False | +0.0 | +2 | +0.707107 | +-0.707107 | +2P | +0.7 | +0.470162 | +0.6622 | +Low | +
+
+
+
+
+
+
+In [261]:
+
+
+
+
+
+print("Statistical summary of demand dataset:")
+print(demand_df.describe())
+
+
+
+
+
+
+
+
+
+
+
+Statistical summary of demand dataset: + zone_number status_hour average_occupancy dayofweek_num hour_sin hour_cos zone_occ_mean \ +count 1573.000000 1573.000000 1573.000000 1573.000000 1573.000000 1.573000e+03 1573.000000 +mean 7466.833439 8.990464 0.523273 3.926256 0.423099 1.245005e-01 0.550234 +std 250.260272 6.701224 0.462691 2.239969 0.635700 6.339444e-01 0.277083 +min 7010.000000 0.000000 0.000000 0.000000 -1.000000 -1.000000e+00 0.000000 +25% 7246.000000 5.000000 0.000000 2.000000 0.000000 -5.000000e-01 0.370370 +50% 7452.000000 7.000000 0.500000 5.000000 0.707107 6.123234e-17 0.571429 +75% 7644.000000 12.000000 1.000000 6.000000 0.965926 7.071068e-01 0.769231 +max 7995.000000 23.000000 1.000000 6.000000 1.000000 1.000000e+00 1.000000 + + zone_occ_std zone_volatility +count 1573.000000 1573.000000 +mean 0.387590 0.836269 +std 0.174669 0.590516 +min 0.000000 0.000000 +25% 0.358569 0.464594 +50% 0.457738 0.774597 +75% 0.500000 1.167712 +max 0.577350 3.201102 ++
+
+
+
+
+
+
+
+In [262]:
+
+
+
+
+
+print("Missing values in cleaned demand_df:")
+print(demand_df.isnull().sum())
+
+
+
+
+
+
+
+
+
+
+
+Missing values in cleaned demand_df: +zone_number 0 +status_day 0 +status_hour 0 +is_weekend 0 +average_occupancy 0 +dayofweek_num 0 +hour_sin 0 +hour_cos 0 +restriction_display 0 +zone_occ_mean 0 +zone_occ_std 0 +zone_volatility 0 +demand_level 0 +dtype: int64 ++
+
+
+
+
+
+
+
+
+Export Processed Data¶
Save the merged bay-level table and the aggregated demand table so they can be reused without repeating the full ingestion and cleaning workflow.
+
+
+
+
+
+
+
+
+In [263]:
+
+
+
+
+
+# Save processed datasets for reuse
+merged_bay_df.to_csv("../data/processed/merged_bay_sensor_data.csv", index=False)
+demand_df.to_csv("../data/processed/cleaned_demand_data.csv", index=False)
+
+print("Files saved successfully.")
+
+
+
+
+
+
+
+
+
+
+
+Files saved successfully. ++
+
+
+
+
+
+
+
+
+Exploratory Data Analysis¶
This section checks the cleaned datasets and visualises demand patterns by hour, day, zone, restriction type, and weekend status. These views help validate whether the engineered features capture realistic parking behaviour before modelling.
+
+
+
+
+
+
+
+
+In [264]:
+
+
+
+
+
+pd.set_option("display.max_columns", None)
+pd.set_option("display.width", 120)
+
+print("Merged dataset preview:")
+display(merged_bay_df.head())
+print("\nMerged dataset info:")
+print(merged_bay_df.info())
+print("\nMissing values in merged_bay_df:")
+print(merged_bay_df.isnull().sum())
+
+print("\nDemand dataset preview:")
+display(demand_df.head())
+print("\nDemand dataset info:")
+print(demand_df.info())
+print("\nMissing values in demand_df:")
+print(demand_df.isnull().sum())
+
+
+
+
+
+
+
+
+
+
+
+Merged dataset preview: ++
+
+
+
+
+
+
+
+
+
+
+| + | status_timestamp | +zone_number | +lastupdated | +kerbsideid | +status_description | +location | +status_hour | +status_day | +status_month | +is_weekend | +occupied | +restriction_display | +time_restrictions_start | +restriction_days | +time_restrictions_finish | +
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | +2024-08-18 08:23:46 | +7394.0 | +2024-12-30 00:44:37 | +9344 | +Unoccupied | +(-37.80494402936792, 144.95916129121264) | +8 | +Sunday | +8 | +True | +0 | +1P | +07:30:00 | +Mon-Sat | +18:30:00 | +
| 1 | +2024-08-14 09:22:05 | +7392.0 | +2025-01-10 02:44:36 | +9373 | +Unoccupied | +(-37.80329074425858, 144.95836336946644) | +9 | +Wednesday | +8 | +False | +0 | +2P | +07:30:00 | +Mon-Fri | +18:30:00 | +
| 2 | +2024-08-14 09:22:05 | +7392.0 | +2025-01-10 02:44:36 | +9373 | +Unoccupied | +(-37.80329074425858, 144.95836336946644) | +9 | +Wednesday | +8 | +False | +0 | +2P | +07:30:00 | +Sat | +12:30:00 | +
| 3 | +2024-11-28 02:30:57 | +7084.0 | +2024-12-04 23:44:37 | +8735 | +Present | +(-37.80223335664963, 144.96120480793184) | +2 | +Thursday | +11 | +False | +1 | +2P | +07:30:00 | +Sat | +12:30:00 | +
| 4 | +2024-11-28 02:30:57 | +7084.0 | +2024-12-04 23:44:37 | +8735 | +Present | +(-37.80223335664963, 144.96120480793184) | +2 | +Thursday | +11 | +False | +1 | +2P | +07:30:00 | +Mon-Fri | +18:30:00 | +
+
+
+
+
++Merged dataset info: +<class 'pandas.DataFrame'> +Index: 7767 entries, 0 to 7995 +Data columns (total 15 columns): + # Column Non-Null Count Dtype +--- ------ -------------- ----- + 0 status_timestamp 7767 non-null datetime64[us] + 1 zone_number 7767 non-null float64 + 2 lastupdated 7767 non-null datetime64[us] + 3 kerbsideid 7767 non-null int64 + 4 status_description 7767 non-null str + 5 location 7767 non-null object + 6 status_hour 7767 non-null int32 + 7 status_day 7767 non-null str + 8 status_month 7767 non-null int32 + 9 is_weekend 7767 non-null bool + 10 occupied 7767 non-null int64 + 11 restriction_display 7767 non-null str + 12 time_restrictions_start 7767 non-null str + 13 restriction_days 7767 non-null str + 14 time_restrictions_finish 7767 non-null str +dtypes: bool(1), datetime64[us](2), float64(1), int32(2), int64(2), object(1), str(6) +memory usage: 857.1+ KB +None + +Missing values in merged_bay_df: +status_timestamp 0 +zone_number 0 +lastupdated 0 +kerbsideid 0 +status_description 0 +location 0 +status_hour 0 +status_day 0 +status_month 0 +is_weekend 0 +occupied 0 +restriction_display 0 +time_restrictions_start 0 +restriction_days 0 +time_restrictions_finish 0 +dtype: int64 + +Demand dataset preview: ++
+
+
+
+
+
+
+
+
+
+
+| + | zone_number | +status_day | +status_hour | +is_weekend | +average_occupancy | +dayofweek_num | +hour_sin | +hour_cos | +restriction_display | +zone_occ_mean | +zone_occ_std | +zone_volatility | +demand_level | +
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | +7010.0 | +Friday | +1 | +False | +0.0 | +4 | +0.258819 | +0.965926 | +2P | +0.7 | +0.470162 | +0.6622 | +Low | +
| 1 | +7010.0 | +Thursday | +21 | +False | +1.0 | +3 | +-0.707107 | +0.707107 | +2P | +0.7 | +0.470162 | +0.6622 | +High | +
| 2 | +7010.0 | +Thursday | +22 | +False | +1.0 | +3 | +-0.500000 | +0.866025 | +2P | +0.7 | +0.470162 | +0.6622 | +High | +
| 3 | +7010.0 | +Thursday | +23 | +False | +1.0 | +3 | +-0.258819 | +0.965926 | +2P | +0.7 | +0.470162 | +0.6622 | +High | +
| 4 | +7010.0 | +Wednesday | +9 | +False | +0.0 | +2 | +0.707107 | +-0.707107 | +2P | +0.7 | +0.470162 | +0.6622 | +Low | +
+
+
+
+
++Demand dataset info: +<class 'pandas.DataFrame'> +RangeIndex: 1573 entries, 0 to 1572 +Data columns (total 13 columns): + # Column Non-Null Count Dtype +--- ------ -------------- ----- + 0 zone_number 1573 non-null float64 + 1 status_day 1573 non-null str + 2 status_hour 1573 non-null int32 + 3 is_weekend 1573 non-null bool + 4 average_occupancy 1573 non-null float64 + 5 dayofweek_num 1573 non-null int64 + 6 hour_sin 1573 non-null float64 + 7 hour_cos 1573 non-null float64 + 8 restriction_display 1573 non-null str + 9 zone_occ_mean 1573 non-null float64 + 10 zone_occ_std 1573 non-null float64 + 11 zone_volatility 1573 non-null float64 + 12 demand_level 1573 non-null str +dtypes: bool(1), float64(7), int32(1), int64(1), str(3) +memory usage: 143.0 KB +None + +Missing values in demand_df: +zone_number 0 +status_day 0 +status_hour 0 +is_weekend 0 +average_occupancy 0 +dayofweek_num 0 +hour_sin 0 +hour_cos 0 +restriction_display 0 +zone_occ_mean 0 +zone_occ_std 0 +zone_volatility 0 +demand_level 0 +dtype: int64 ++
+
+
+
+
+
+
+
+
+1. Average Parking Demand by Hour¶
+
+
+
+
+
+
+
+
+In [265]:
+
+
+
+
+
+hourly_demand = demand_df.groupby("status_hour", as_index=False)["average_occupancy"].mean()
+display(hourly_demand)
+
+plt.figure(figsize=(10, 4))
+plt.plot(hourly_demand["status_hour"], hourly_demand["average_occupancy"], marker="o")
+plt.title("Average Parking Demand by Hour")
+plt.xlabel("Hour of Day")
+plt.ylabel("Average Occupancy")
+plt.xticks(range(0, 24))
+plt.grid(True)
+plt.show()
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+| + | status_hour | +average_occupancy | +
|---|---|---|
| 0 | +0 | +0.672394 | +
| 1 | +1 | +0.556783 | +
| 2 | +2 | +0.550073 | +
| 3 | +3 | +0.536392 | +
| 4 | +4 | +0.477390 | +
| 5 | +5 | +0.516237 | +
| 6 | +6 | +0.583623 | +
| 7 | +7 | +0.675812 | +
| 8 | +8 | +0.506127 | +
| 9 | +9 | +0.363462 | +
| 10 | +10 | +0.265625 | +
| 11 | +11 | +0.264706 | +
| 12 | +12 | +0.220588 | +
| 13 | +13 | +0.181818 | +
| 14 | +14 | +0.368421 | +
| 15 | +15 | +0.176471 | +
| 16 | +16 | +0.285714 | +
| 17 | +17 | +0.500000 | +
| 18 | +18 | +0.769231 | +
| 19 | +19 | +0.489418 | +
| 20 | +20 | +0.470213 | +
| 21 | +21 | +0.489583 | +
| 22 | +22 | +0.535135 | +
| 23 | +23 | +0.607479 | +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+2. Average Parking Demand by Day of the Week¶
+
+
+
+
+
+
+
+
+In [266]:
+
+
+
+
+
+day_order = ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"]
+daily_demand = demand_df.groupby("status_day", as_index=False)["average_occupancy"].mean()
+daily_demand["status_day"] = pd.Categorical(daily_demand["status_day"], categories=day_order, ordered=True)
+daily_demand = daily_demand.sort_values("status_day")
+display(daily_demand)
+
+plt.figure(figsize=(10, 4))
+plt.bar(daily_demand["status_day"], daily_demand["average_occupancy"])
+plt.title("Average Parking Demand by Day of the Week")
+plt.xlabel("Day of the Week")
+plt.ylabel("Average Occupancy")
+plt.xticks(rotation=45)
+plt.show()
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+| + | status_day | +average_occupancy | +
|---|---|---|
| 1 | +Monday | +0.537356 | +
| 5 | +Tuesday | +0.460080 | +
| 6 | +Wednesday | +0.539483 | +
| 4 | +Thursday | +0.409084 | +
| 0 | +Friday | +0.504430 | +
| 2 | +Saturday | +0.432759 | +
| 3 | +Sunday | +0.578658 | +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+3. Demand Level Distribution¶
+
+
+
+
+
+
+
+
+In [267]:
+
+
+
+
+
+print(demand_df["demand_level"].value_counts())
+
+plt.figure(figsize=(8, 4))
+demand_df["demand_level"].value_counts().plot(kind="bar")
+plt.title("Distribution of Parking Demand Levels")
+plt.xlabel("Demand Level")
+plt.ylabel("Count")
+plt.show()
+
+
+
+
+
+
+
+
+
+
+
+demand_level +High 767 +Low 649 +Medium 157 +Name: count, dtype: int64 ++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+4. Zone-Based Parking Demand¶
+
+
+
+
+
+
+
+
+In [268]:
+
+
+
+
+
+zone_summary = (
+ merged_bay_df.groupby("zone_number")
+ .agg(average_occupancy=("occupied", "mean"), observation_count=("occupied", "count"))
+ .reset_index()
+ .sort_values("average_occupancy", ascending=False)
+)
+
+display(zone_summary.head(10))
+
+plt.figure(figsize=(12, 4))
+plt.bar(zone_summary.head(10)["zone_number"].astype(str), zone_summary.head(10)["average_occupancy"])
+plt.title("Top 10 Parking Zones by Average Occupancy")
+plt.xlabel("Zone Number")
+plt.ylabel("Average Occupancy")
+plt.xticks(rotation=45)
+plt.show()
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+| + | zone_number | +average_occupancy | +observation_count | +
|---|---|---|---|
| 2 | +7014.0 | +1.0 | +8 | +
| 303 | +7765.0 | +1.0 | +8 | +
| 24 | +7182.0 | +1.0 | +3 | +
| 27 | +7185.0 | +1.0 | +3 | +
| 30 | +7189.0 | +1.0 | +9 | +
| 292 | +7728.0 | +1.0 | +10 | +
| 295 | +7740.0 | +1.0 | +10 | +
| 299 | +7757.0 | +1.0 | +32 | +
| 280 | +7706.0 | +1.0 | +4 | +
| 65 | +7244.0 | +1.0 | +2 | +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+5. Parking Restrictions and Weekend Patterns¶
+
+
+
+
+
+
+
+
+In [269]:
+
+
+
+
+
+restriction_demand = (
+ merged_bay_df.groupby("restriction_display", as_index=False)["occupied"]
+ .mean()
+ .sort_values("occupied", ascending=False)
+)
+weekend_comparison = merged_bay_df.groupby("is_weekend", as_index=False)["occupied"].mean()
+hour_weekend_demand = demand_df.groupby(["status_hour", "is_weekend"], as_index=False)["average_occupancy"].mean()
+
+display(restriction_demand.head(10))
+plt.figure(figsize=(12, 4))
+plt.bar(restriction_demand.head(10)["restriction_display"], restriction_demand.head(10)["occupied"])
+plt.title("Average Occupancy by Restriction Display (Top 10)")
+plt.xlabel("Restriction Display")
+plt.ylabel("Average Occupancy")
+plt.xticks(rotation=45)
+plt.show()
+
+display(weekend_comparison)
+plt.figure(figsize=(6, 4))
+plt.bar(weekend_comparison["is_weekend"].astype(str), weekend_comparison["occupied"])
+plt.title("Average Occupancy: Weekday vs Weekend")
+plt.xlabel("Is Weekend")
+plt.ylabel("Average Occupancy")
+plt.show()
+
+display(hour_weekend_demand.head())
+plt.figure(figsize=(10, 4))
+for weekend_value in hour_weekend_demand["is_weekend"].unique():
+ subset = hour_weekend_demand[hour_weekend_demand["is_weekend"] == weekend_value]
+ plt.plot(subset["status_hour"], subset["average_occupancy"], marker="o", label=f"is_weekend={weekend_value}")
+plt.legend()
+plt.title("Hourly Demand by Weekend Status")
+plt.xlabel("Hour")
+plt.ylabel("Average Occupancy")
+plt.show()
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+| + | restriction_display | +occupied | +
|---|---|---|
| 6 | +HP | +1.000000 | +
| 4 | +FP1P | +0.823529 | +
| 9 | +MP1P | +0.791667 | +
| 14 | +Unknown | +0.666667 | +
| 0 | +1P | +0.653944 | +
| 10 | +MP2P | +0.601074 | +
| 8 | +LZ30 | +0.570698 | +
| 13 | +SP | +0.545455 | +
| 3 | +DP2P | +0.545455 | +
| 1 | +2P | +0.538825 | +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+| + | is_weekend | +occupied | +
|---|---|---|
| 0 | +False | +0.537645 | +
| 1 | +True | +0.610114 | +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+| + | status_hour | +is_weekend | +average_occupancy | +
|---|---|---|---|
| 0 | +0 | +False | +0.679693 | +
| 1 | +0 | +True | +0.647059 | +
| 2 | +1 | +False | +0.505952 | +
| 3 | +1 | +True | +0.673913 | +
| 4 | +2 | +False | +0.468869 | +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+6. Statistical Summary¶
+
+
+
+
+
+
+
+
+In [270]:
+
+
+
+
+
+print("Statistical summary of demand dataset:")
+print(demand_df.describe())
+print("\nCorrelation summary:")
+print(demand_df[["zone_number", "status_hour", "is_weekend", "average_occupancy"]].corr(numeric_only=True))
+
+
+
+
+
+
+
+
+
+
+
+Statistical summary of demand dataset: + zone_number status_hour average_occupancy dayofweek_num hour_sin hour_cos zone_occ_mean \ +count 1573.000000 1573.000000 1573.000000 1573.000000 1573.000000 1.573000e+03 1573.000000 +mean 7466.833439 8.990464 0.523273 3.926256 0.423099 1.245005e-01 0.550234 +std 250.260272 6.701224 0.462691 2.239969 0.635700 6.339444e-01 0.277083 +min 7010.000000 0.000000 0.000000 0.000000 -1.000000 -1.000000e+00 0.000000 +25% 7246.000000 5.000000 0.000000 2.000000 0.000000 -5.000000e-01 0.370370 +50% 7452.000000 7.000000 0.500000 5.000000 0.707107 6.123234e-17 0.571429 +75% 7644.000000 12.000000 1.000000 6.000000 0.965926 7.071068e-01 0.769231 +max 7995.000000 23.000000 1.000000 6.000000 1.000000 1.000000e+00 1.000000 + + zone_occ_std zone_volatility +count 1573.000000 1573.000000 +mean 0.387590 0.836269 +std 0.174669 0.590516 +min 0.000000 0.000000 +25% 0.358569 0.464594 +50% 0.457738 0.774597 +75% 0.500000 1.167712 +max 0.577350 3.201102 + +Correlation summary: + zone_number status_hour is_weekend average_occupancy +zone_number 1.000000 -0.076096 0.120871 -0.006406 +status_hour -0.076096 1.000000 -0.135417 -0.056190 +is_weekend 0.120871 -0.135417 1.000000 0.069418 +average_occupancy -0.006406 -0.056190 0.069418 1.000000 ++
+
+
+
+
+
+
+
+
+
+Machine Learning Model Development¶
The target variable is demand_level, which classifies average occupancy as low, medium, or high. The models use zone, hour, day, and weekend indicators to compare a simple linear baseline with a more flexible Random Forest classifier.
+
+
+
+
+
+
+
+
+1. Feature Matrix and Train-Test Split¶
The model uses spatial and temporal fields as predictors and demand_level as the target. Stratification keeps the low, medium, and high demand classes represented in both training and test sets.
+
+
+
+
+
+
+
+In [271]:
+
+
+
+
+
+numeric_features = [
+ "average_occupancy",
+ "dayofweek_num",
+ "is_weekend",
+ "hour_sin",
+ "hour_cos",
+ "zone_occ_mean",
+ "zone_occ_std",
+ "zone_volatility",
+]
+categorical_features = ["status_day", "restriction_display"]
+
+X = demand_df[[
+ "average_occupancy",
+ "hour_sin",
+ "hour_cos",
+ "status_day",
+ "dayofweek_num",
+ "is_weekend",
+ "restriction_display",
+ "zone_occ_mean",
+ "zone_occ_std",
+ "zone_volatility",
+]]
+y = demand_df["demand_level"]
+
+X_train, X_test, y_train, y_test = train_test_split(
+ X,
+ y,
+ test_size=0.2,
+ random_state=42,
+ stratify=y,
+)
+
+print("Feature sample:")
+display(X.head())
+print("\nTarget sample:")
+display(y.head())
+print("\nClass distribution in training set:")
+print(y_train.value_counts())
+print("\nClass distribution in test set:")
+print(y_test.value_counts())
+print("\nTraining set size:", X_train.shape)
+print("Testing set size:", X_test.shape)
+
+
+
+
+
+
+
+
+
+
+
+Feature sample: ++
+
+
+
+
+
+
+
+
+
+
+| + | average_occupancy | +hour_sin | +hour_cos | +status_day | +dayofweek_num | +is_weekend | +restriction_display | +zone_occ_mean | +zone_occ_std | +zone_volatility | +
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | +0.0 | +0.258819 | +0.965926 | +Friday | +4 | +False | +2P | +0.7 | +0.470162 | +0.6622 | +
| 1 | +1.0 | +-0.707107 | +0.707107 | +Thursday | +3 | +False | +2P | +0.7 | +0.470162 | +0.6622 | +
| 2 | +1.0 | +-0.500000 | +0.866025 | +Thursday | +3 | +False | +2P | +0.7 | +0.470162 | +0.6622 | +
| 3 | +1.0 | +-0.258819 | +0.965926 | +Thursday | +3 | +False | +2P | +0.7 | +0.470162 | +0.6622 | +
| 4 | +0.0 | +0.707107 | +-0.707107 | +Wednesday | +2 | +False | +2P | +0.7 | +0.470162 | +0.6622 | +
+
+
+
+
++Target sample: ++
+
+
+
+
+0 Low +1 High +2 High +3 High +4 Low +Name: demand_level, dtype: str+
+
+
+
+
++Class distribution in training set: +demand_level +High 613 +Low 519 +Medium 126 +Name: count, dtype: int64 + +Class distribution in test set: +demand_level +High 154 +Low 130 +Medium 31 +Name: count, dtype: int64 + +Training set size: (1258, 10) +Testing set size: (315, 10) ++
+
+
+
+
+
+
+
+
+2. Preprocessing Pipeline¶
Numeric features are imputed and scaled, while categorical features are one-hot encoded. The shared preprocessor keeps the logistic regression and Random Forest pipelines directly comparable.
+
+
+
+
+
+
+
+
+In [272]:
+
+
+
+
+
+numeric_features = ["average_occupancy", "hour_sin", "hour_cos", "dayofweek_num",
+ "zone_occ_mean", "zone_occ_std", "zone_volatility"]
+categorical_features = ["status_day", "restriction_display"]
+
+numeric_transformer = Pipeline(steps=[
+ ("imputer", SimpleImputer(strategy="median")),
+ ("scaler", StandardScaler()),
+])
+
+categorical_transformer = Pipeline(steps=[
+ ("onehot", OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
+])
+
+preprocessor = ColumnTransformer(
+ transformers=[
+ ("num", numeric_transformer, numeric_features),
+ ("cat", categorical_transformer, categorical_features),
+ ]
+)
+
+
+
+
+
+
+
+
+
+2. Logistic Regression Baseline¶
+
+
+
+
+
+
+
+
+In [273]:
+
+
+
+
+
+logistic_model = Pipeline(steps=[
+ ("preprocessor", preprocessor),
+ ("classifier", LogisticRegression(max_iter=3000, class_weight="balanced")),
+])
+logistic_model.fit(X_train, y_train)
+y_pred_logistic = logistic_model.predict(X_test)
+
+print("Logistic Regression (balanced class weights) classification report:")
+print(classification_report(y_test, y_pred_logistic, zero_division=0))
+
+cm_logistic = confusion_matrix(y_test, y_pred_logistic, labels=logistic_model.classes_)
+disp = ConfusionMatrixDisplay(confusion_matrix=cm_logistic, display_labels=logistic_model.classes_)
+disp.plot()
+plt.title("Confusion Matrix - Logistic Regression")
+plt.show()
+
+# Get probability predictions for threshold tuning
+y_pred_proba_logistic = logistic_model.predict_proba(X_test)
+
+
+
+
+
+
+
+
+
+
+
+Logistic Regression (balanced class weights) classification report: + precision recall f1-score support + + High 1.00 0.99 0.99 154 + Low 1.00 0.98 0.99 130 + Medium 0.86 1.00 0.93 31 + + accuracy 0.98 315 + macro avg 0.95 0.99 0.97 315 +weighted avg 0.99 0.98 0.98 315 + ++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+3. Random Forest Model¶
+
+
+
+
+
+
+
+
+In [274]:
+
+
+
+
+
+random_forest_model = Pipeline(steps=[
+ ("preprocessor", preprocessor),
+ ("classifier", RandomForestClassifier(
+ n_estimators=300,
+ min_samples_leaf=3,
+ max_depth=15,
+ class_weight="balanced_subsample",
+ random_state=42,
+ n_jobs=-1,
+ )),
+])
+random_forest_model.fit(X_train, y_train)
+y_pred_rf = random_forest_model.predict(X_test)
+
+print("Random Forest (balanced class weights) classification report:")
+print(classification_report(y_test, y_pred_rf, zero_division=0))
+
+cm_rf = confusion_matrix(y_test, y_pred_rf, labels=random_forest_model.classes_)
+disp = ConfusionMatrixDisplay(confusion_matrix=cm_rf, display_labels=random_forest_model.classes_)
+disp.plot()
+plt.title("Confusion Matrix - Random Forest")
+plt.show()
+
+# Feature importance
+try:
+ importances = random_forest_model.named_steps["classifier"].feature_importances_
+ # Get feature names from preprocessor
+ cat_features_ohe = list(random_forest_model.named_steps["preprocessor"]
+ .named_transformers_['cat']
+ .named_steps['onehot']
+ .get_feature_names_out(categorical_features))
+ all_feature_names = numeric_features + cat_features_ohe
+
+ top_indices = np.argsort(importances)[-10:]
+ plt.figure(figsize=(10, 6))
+ plt.barh(range(len(top_indices)), importances[top_indices])
+ plt.yticks(range(len(top_indices)), [str(all_feature_names[i])[:30] for i in top_indices])
+ plt.xlabel("Importance")
+ plt.title("Top 10 Feature Importances - Random Forest")
+ plt.tight_layout()
+ plt.show()
+except Exception as e:
+ print(f"Note: Could not extract feature importance: {e}")
+
+# Get probability predictions for threshold tuning
+y_pred_proba_rf = random_forest_model.predict_proba(X_test)
+
+
+
+
+
+
+
+
+
+
+
+Random Forest (balanced class weights) classification report: + precision recall f1-score support + + High 1.00 1.00 1.00 154 + Low 1.00 1.00 1.00 130 + Medium 1.00 1.00 1.00 31 + + accuracy 1.00 315 + macro avg 1.00 1.00 1.00 315 +weighted avg 1.00 1.00 1.00 315 + ++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+4. Model Comparison and Cross-Validation¶
+
+
+
+
+
+
+
+
+In [275]:
+
+
+
+
+
+comparison_df = pd.DataFrame({
+ "Model": ["Logistic Regression", "Random Forest"],
+ "Accuracy": [accuracy_score(y_test, y_pred_logistic), accuracy_score(y_test, y_pred_rf)],
+ "Balanced Accuracy": [balanced_accuracy_score(y_test, y_pred_logistic), balanced_accuracy_score(y_test, y_pred_rf)],
+ "Macro F1": [f1_score(y_test, y_pred_logistic, average="macro"), f1_score(y_test, y_pred_rf, average="macro")],
+})
+
+display(comparison_df.round(4))
+
+plot_df = comparison_df.set_index("Model")[["Balanced Accuracy", "Macro F1"]]
+plt.figure(figsize=(9, 5))
+plot_df.plot(kind="bar", ylim=(0, 1), rot=0)
+plt.title("Model Comparison: Balanced Accuracy and Macro F1")
+plt.ylabel("Score")
+plt.xlabel("Model")
+plt.legend(loc="lower right")
+plt.tight_layout()
+plt.show()
+
+cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
+cv_scores = cross_validate(
+ random_forest_model,
+ X,
+ y,
+ cv=cv,
+ scoring={
+ "accuracy": "accuracy",
+ "balanced_accuracy": "balanced_accuracy",
+ "macro_f1": "f1_macro",
+ },
+ n_jobs=-1,
+)
+
+cv_summary = pd.DataFrame({
+ "Metric": ["Accuracy", "Balanced Accuracy", "Macro F1"],
+ "Mean": [
+ cv_scores["test_accuracy"].mean(),
+ cv_scores["test_balanced_accuracy"].mean(),
+ cv_scores["test_macro_f1"].mean(),
+ ],
+ "Std": [
+ cv_scores["test_accuracy"].std(),
+ cv_scores["test_balanced_accuracy"].std(),
+ cv_scores["test_macro_f1"].std(),
+ ],
+})
+
+display(cv_summary.round(4))
+
+print("\n✓ Improvements applied:")
+print(" - Zone-level features (occupancy variance, duration stats)")
+print(" - Balanced class weights in both models")
+print(" - Hyperparameter tuning (max_depth=15, min_samples_leaf=3)")
+print(" - Feature importance analysis")
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+| + | Model | +Accuracy | +Balanced Accuracy | +Macro F1 | +
|---|---|---|---|---|
| 0 | +Logistic Regression | +0.9841 | +0.988 | +0.9691 | +
| 1 | +Random Forest | +1.0000 | +1.000 | +1.0000 | +
+
+
+
+
+<Figure size 900x500 with 0 Axes>+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+| + | Metric | +Mean | +Std | +
|---|---|---|---|
| 0 | +Accuracy | +0.9962 | +0.0037 | +
| 1 | +Balanced Accuracy | +0.9871 | +0.0125 | +
| 2 | +Macro F1 | +0.9919 | +0.0079 | +
+
+
+
+
++✓ Improvements applied: + - Zone-level features (occupancy variance, duration stats) + - Balanced class weights in both models + - Hyperparameter tuning (max_depth=15, min_samples_leaf=3) + - Feature importance analysis ++
+
+
+
+
+
+
+
+
+Temporal Holdout¶
A date-based split tests whether the model can generalise from earlier parking observations to later observations. This is closer to a real forecasting workflow than a random split because future data is not allowed to leak into training.
+
+
+
+
+
+
+
+
+In [276]:
+
+
+
+
+
+# Build daily demand per zone (requires `merged_bay_df` with `status_timestamp` and `zone_number`)
+daily = merged_bay_df.copy()
+if "status_timestamp" not in daily.columns:
+ raise RuntimeError("status_timestamp not found in merged_bay_df — cannot build temporal holdout")
+
+daily["date"] = pd.to_datetime(daily["status_timestamp"]).dt.date
+daily_zone = (
+ daily.groupby(["zone_number", "date"], as_index=False)["occupied"].mean().rename(columns={"occupied": "average_occupancy"})
+)
+
+def demand_level_val(v):
+ if v < 0.33:
+ return "Low"
+ elif v < 0.66:
+ return "Medium"
+ else:
+ return "High"
+
+daily_zone["demand_level"] = daily_zone["average_occupancy"].apply(demand_level_val)
+
+# Attach zone metadata for interpretation
+zone_meta = merged_bay_df.groupby("zone_number", as_index=False).agg({"restriction_display": "first", "location": "first"})
+daily_zone = daily_zone.merge(zone_meta, on="zone_number", how="left")
+
+# Add daily temporal features
+daily_zone["dayofweek"] = pd.to_datetime(daily_zone["date"]).dt.day_name()
+daily_zone["month"] = pd.to_datetime(daily_zone["date"]).dt.month
+
+X_cols = ["zone_number", "dayofweek", "month", "restriction_display"]
+y_col = "demand_level"
+
+# Train on the earliest dates and test on later dates
+dates = sorted(daily_zone["date"].unique())
+cutoff = dates[int(len(dates) * 0.8)] if len(dates) > 5 else dates[int(len(dates) * 0.6)]
+train_mask = daily_zone["date"] <= cutoff
+train_df = daily_zone[train_mask].copy()
+test_df = daily_zone[~train_mask].copy()
+
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import OneHotEncoder
+from sklearn.compose import ColumnTransformer
+
+num_feats = ["zone_number", "month"]
+cat_feats = ["dayofweek", "restriction_display"]
+
+num_transformer = Pipeline([("imputer", SimpleImputer(strategy="median")), ("scaler", StandardScaler())])
+cat_transformer = Pipeline([("onehot", OneHotEncoder(handle_unknown="ignore"))])
+
+preproc_daily = ColumnTransformer([("num", num_transformer, num_feats), ("cat", cat_transformer, cat_feats)])
+
+rf_daily = Pipeline([("preproc", preproc_daily), ("clf", RandomForestClassifier(n_estimators=200, random_state=42, class_weight="balanced_subsample"))])
+
+rf_daily.fit(train_df[X_cols], train_df[y_col])
+pred = rf_daily.predict(test_df[X_cols])
+
+from sklearn.metrics import classification_report
+print("Temporal holdout — overall classification report (test period):")
+print(classification_report(test_df[y_col], pred, zero_division=0))
+
+# Per-zone performance (only include zones with >= min_samples in the test set)
+zone_metrics = []
+min_samples = 10
+for z, group in test_df.groupby("zone_number"):
+ if len(group) < min_samples:
+ continue
+ y_true = group[y_col]
+ y_pred = rf_daily.predict(group[X_cols])
+ rep = classification_report(y_true, y_pred, output_dict=True, zero_division=0)
+ # Keep summary metrics that are easier to compare across zones
+ zone_metrics.append({
+ "zone_number": z,
+ "n_samples": len(group),
+ "macro_f1": rep.get("macro avg", {}).get("f1-score", None),
+ "weighted_f1": rep.get("weighted avg", {}).get("f1-score", None),
+ })
+
+zone_metrics_df = pd.DataFrame(zone_metrics).sort_values("macro_f1", ascending=False)
+display(zone_metrics_df.head(20))
+# Save per-zone metrics for later review
+zone_metrics_df.to_csv("../data/processed/per_zone_holdout_metrics.csv", index=False)
+print("Per-zone metrics saved to ../data/processed/per_zone_holdout_metrics.csv")
+
+
+
+
+
+
+
+
+
+
+
+Temporal holdout — overall classification report (test period): + precision recall f1-score support + + High 0.54 0.68 0.60 216 + Low 0.44 0.44 0.44 136 + Medium 0.00 0.00 0.00 61 + + accuracy 0.50 413 + macro avg 0.32 0.37 0.35 413 +weighted avg 0.42 0.50 0.46 413 + ++
+
+
+
+
+
+
+
+
+
+
+| + | zone_number | +n_samples | +macro_f1 | +weighted_f1 | +
|---|---|---|---|---|
| 1 | +7186.0 | +10 | +0.583333 | +0.583333 | +
| 0 | +7173.0 | +13 | +0.372294 | +0.515485 | +
+
+
+
+
+Per-zone metrics saved to ../data/processed/per_zone_holdout_metrics.csv ++
+
+
+
+
+
+
+
+
+High-Demand Zone Map¶
The map highlights the highest-occupancy zones when latitude and longitude are available. Marker size represents the number of observations, and marker colour represents average occupancy.
+
+
+
+
+
+
+
+
+In [277]:
+
+
+
+
+
+import folium
+
+# extract lat/lon from `location` (supports tuple/list or dict shapes)
+def extract_latlon(v):
+ if isinstance(v, (list, tuple)) and len(v) >= 2:
+ return float(v[0]), float(v[1])
+ if isinstance(v, dict):
+ # some datasets use {'latitude':..,'longitude':..}
+ if "latitude" in v and "longitude" in v:
+ return float(v["latitude"]), float(v["longitude"])
+ if "lat" in v and "lon" in v:
+ return float(v["lat"]), float(v["lon"])
+ return None, None
+
+zone_meta[["lat", "lon"]] = zone_meta["location"].apply(lambda v: pd.Series(extract_latlon(v)))
+
+# top zones by average occupancy (overall)
+top_zones = zone_summary.head(20).merge(zone_meta, on="zone_number", how="left")
+top_zones = top_zones.dropna(subset=["lat", "lon"]).copy()
+
+if top_zones.shape[0] == 0:
+ print("No lat/lon available for top zones; map cannot be built")
+else:
+ center_lat = top_zones["lat"].mean()
+ center_lon = top_zones["lon"].mean()
+ m = folium.Map(location=[center_lat, center_lon], zoom_start=13)
+
+ for _, r in top_zones.iterrows():
+ popup = folium.Popup(f"Zone {r['zone_number']}<br>Avg occ: {r['average_occupancy']:.2f}<br>Restriction: {r.get('restriction_display','')}", max_width=300)
+ folium.CircleMarker(location=[r["lat"], r["lon"]], radius=6, popup=popup, color="crimson", fill=True).add_to(m)
+
+ display(m)
+
+
+
+
+
+
+
+
+
+
+
+
+Make this Notebook Trusted to load map: File -> Trust Notebook
+
+
+
+
+
+
+
+
+Example Prediction¶
The final example sends one parking-zone scenario through both trained models to show how the notebook can be used for a single demand classification.
+
+
+
+
+
+
+
+
+In [278]:
+
+
+
+
+
+# Look up zone features from the training data
+zone_num = 7303
+zone_data = demand_df[demand_df["zone_number"] == zone_num]
+
+# Use zone-specific stats if available, otherwise use dataset mean
+if zone_data.shape[0] > 0:
+ zone_occ_mean = zone_data["zone_occ_mean"].iloc[0]
+ zone_occ_std = zone_data["zone_occ_std"].iloc[0]
+ zone_volatility = zone_data["zone_volatility"].iloc[0]
+else:
+ # Fallback to dataset means if zone not in training data
+ zone_occ_mean = demand_df["zone_occ_mean"].mean()
+ zone_occ_std = demand_df["zone_occ_std"].mean()
+ zone_volatility = demand_df["zone_volatility"].mean()
+
+sample_input = pd.DataFrame({
+ "average_occupancy": [0.45],
+ "hour_sin": [np.sin(2 * np.pi * 10 / 24)],
+ "hour_cos": [np.cos(2 * np.pi * 10 / 24)],
+ "status_day": ["Monday"],
+ "dayofweek_num": [0],
+ "is_weekend": [False],
+ "restriction_display": ["2P"],
+ "zone_occ_mean": [zone_occ_mean],
+ "zone_occ_std": [zone_occ_std],
+ "zone_volatility": [zone_volatility],
+})
+
+print("Example prediction:")
+display(sample_input)
+print("Logistic Regression Prediction:", logistic_model.predict(sample_input)[0])
+print("Random Forest Prediction:", random_forest_model.predict(sample_input)[0])
+print("Random Forest class probabilities:")
+display(pd.DataFrame(random_forest_model.predict_proba(sample_input), columns=random_forest_model.classes_).round(3))
+
+
+
+
+
+
+
+
+
+
+
+Example prediction: ++
+
+
+
+
+
+
+
+
+
+
+| + | average_occupancy | +hour_sin | +hour_cos | +status_day | +dayofweek_num | +is_weekend | +restriction_display | +zone_occ_mean | +zone_occ_std | +zone_volatility | +
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | +0.45 | +0.5 | +-0.866025 | +Monday | +0 | +False | +2P | +0.666667 | +0.57735 | +0.853227 | +
+
+
+
+
+Logistic Regression Prediction: Medium +Random Forest Prediction: Medium +Random Forest class probabilities: ++
+
+
+
+
+
+
+
+
+
+
+| + | High | +Low | +Medium | +
|---|---|---|---|
| 0 | +0.201 | +0.313 | +0.486 | +
+
+
+
+
+
+
+
+
+
+The cleaned dataset now represents average parking occupancy by zone, day, hour, and weekend status. This makes it suitable for later exploratory data analysis and for building a model to classify parking demand as low, medium, or high.
+
+
+
+
+
+
+
+
+
+
+Conclusions, Recommendations, and Next Steps¶
-
+
The notebook shows that parking demand can be modelled effectively using a combination of temporal, spatial, and restriction-based features.
+
+Both Logistic Regression and Random Forest capture useful structure in the data, and the final pipeline is suitable as a strong baseline for this use case.
+
+For future improvement, consider adding external context such as weather, events, or live traffic signals, and validate the workflow with a temporal holdout to better reflect real-world deployment.
+
+