updated solution for the py-modindex.html documentation problem

.gitignore

@@ -6,6 +6,8 @@
 !*file
 !demo/demo_cli.ipynb
 !demo/demo_python.ipynb
+!demo/demo_datatree.ipynb
+!*.py.old
 
 __pycache__/
 *.egg-info/

demo/demo_datatree.ipynb

@@ -0,0 +1,360 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "77d8504a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pathlib import Path\n",
+    "from hallmark.eht_datatree import build_tree\n",
+    "from hallmark.fmt_detection import scan_inventory\n",
+    "import pandas as pd\n",
+    "pd.set_option(\"display.width\", 300)\n",
+    "pd.set_option(\"display.max_columns\", None)\n",
+    "pd.set_option(\"display.max_colwidth\", 50)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7bb3a37b",
+   "metadata": {},
+   "source": [
+    "a dataree can be built based on the path to the root and the data's fmts. Multiple fmts can be inputted. If no fmts are inputted, the program will try to detect them from the file name patterns, but this functionality still isn't perfect. Flag for if the data is L1 or L2 as the two trees are built slightly different"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "91c2e301",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "datasets = [\"EHTC_CenA2017_Jul2021\", \"EHTC_M87-2018_Mar2024\",\"EHTC_SgrAmwl2017_May2022\",\n",
+    "             \"EHTC_First3C279Results_May2020\", \"EHTC_M87mwl2017_Apr2021\",      \n",
+    "             \"EHTC_metadata2018_Dec2023\", \"EHTC_FirstM87Results_Apr2019\",\n",
+    "             \"EHTC_M87mwl2018_Dec2024\", \"EHTC_FirstSgrAPol_Mar2024\",\n",
+    "             \"EHTC_M87pol2017_Nov2023\", \"EHTC_FirstSgrAResults_May2022\",\n",
+    "             \"EHTC_MonitoringM87_Sep2020\"]\n",
+    "root = Path(f\"~/{datasets[10]}\").expanduser()\n",
+    "tree = build_tree(root, None, data_type=\"L2\")\n",
+    "print(\"tree keys:\", list(tree.keys()))\n",
+    "#files = scan_inventory(root)\n",
+    "#ames = sorted(Path(f).name for f in files)\n",
+    "#for name in names:\n",
+    "#   print(name)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "766ae073",
+   "metadata": {},
+   "source": [
+    "\"meta\" contains all the housekeeping files, which will be any files besides drives or files that have fmt naming conventions"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0cb26601",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(\"=== META ===\")\n",
+    "print(tree[\"meta\"])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c1b7eb90",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if \"drives\" in tree:\n",
+    "    print(\"=== DRIVES ===\")\n",
+    "    print(tree[\"drives\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4e6d7164",
+   "metadata": {},
+   "source": [
+    "\"data\" will be a nested dict that has an outer list of all the different fmts"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "dbc50055",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(\"=== DATA FMTS ===\")\n",
+    "fmt_keys = list(tree[\"data\"].keys())\n",
+    "for stem in fmt_keys:\n",
+    "    print(f\"{stem}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e961e8bc",
+   "metadata": {},
+   "source": [
+    "each fmt has a different Paraframe for each stem, as well as a Paraframe of all the stems merged incase someone wants to view all the data for one fmt"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d3747445",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(f\"=== {fmt_keys[0]} STEMS ===\")\n",
+    "stem_keys = list(tree[\"data\"][fmt_keys[0]].keys())\n",
+    "for key in tree[\"data\"][fmt_keys[0]].keys():\n",
+    "    print(key)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "41e091d0",
+   "metadata": {},
+   "source": [
+    "files that share fmts besides a missing parameter will also be grouped with that fmt, despite technically being parsed with a seperate fmt."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9a270ad4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(f\"=== ALL PARAFRAME  ===\")\n",
+    "print(tree[\"data\"][fmt_keys[0]][\"all\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "dfc324ef",
+   "metadata": {},
+   "source": [
+    "The invindual stem Paraframes contain the same data with the different file formats"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "866848c8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(f\"=== {stem_keys[0]} STEM ===\")\n",
+    "print(\n",
+    "tree[\"data\"][fmt_keys[0]][stem_keys[0]])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ee06d6c6",
+   "metadata": {},
+   "source": [
+    "The merged paraframe can be filtered to produce the same output as any given stem"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5c0ee919",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(\"=== MERGED PARAFRAME FILTERED ===\")\n",
+    "pf = tree[\"data\"][fmt_keys[0]][\"all\"]\n",
+    "\n",
+    "# get cols and params to ease testing and avoid hardcoding values\n",
+    "param_columns = [column for column in pf.columns if column not in (\"path\", \"ext\")]\n",
+    "filter_kwargs = {col: pf.iloc[0][col] for col in param_columns}\n",
+    "for column, value in filter_kwargs.items():\n",
+    "    pf = pf(**{column: value})\n",
+    "    \n",
+    "# can also print by doing print(pf(column0=\"value0\")(column1=\"value1\"),...)\n",
+    "print(pf)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1d58a4af",
+   "metadata": {},
+   "source": [
+    "An L1 datatree is built similarly to the L2 case. Each zip drive is treated as its own L2 dataset, and prior to that will rely on recursive build_tree calls creating new branches for each folder until the zip drives are hit"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "19e53ab2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "root = Path(\"~/eht_local_copy\").expanduser()\n",
+    "drive_fmts = [\n",
+    "    \"e17a10-7-hi-{p0}.{p1}\",\n",
+    "    \"e17b06-7-hi_{p2}.{p3}\",\n",
+    "    \"ff-{p0}-{p1}-{p2}-{p3}-{p4}.{p5}\",\n",
+    "    \"{p0}.B.{p1}.{p2}\",]\n",
+    "tree = build_tree(root, drive_fmts, data_type=\"L1\")\n",
+    "root_keys = list(tree.keys())\n",
+    "print(\"=== ROOT KEYS ===\")\n",
+    "print(root_keys)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a217d6bc",
+   "metadata": {},
+   "source": [
+    "No \"data\" branches will be made until we reach a folder that has compressed drives"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0f76048a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "all_project_keys = {}\n",
+    "for key in root_keys:\n",
+    "    if key == \"meta\":\n",
+    "        print(\"=== META PARAFRAME ===\")\n",
+    "        print(tree[key])\n",
+    "        print()\n",
+    "    elif key == \"drives\":\n",
+    "        print(\"=== DRIVES PARAFRAME ===\")\n",
+    "        print(tree[key])\n",
+    "        print()\n",
+    "    else:\n",
+    "        project_keys = list(tree[key].keys())\n",
+    "        all_project_keys[key] = project_keys\n",
+    "        print(f\"=== {key} KEYS ===\")\n",
+    "        for key in project_keys:\n",
+    "            print(key)\n",
+    "        print()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "14ee04ae",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "all_extract_keys = {}\n",
+    "for project_name in all_project_keys:\n",
+    "    print(f\"=== {project_name} PROJECT ===\")\n",
+    "    for extract_name in all_project_keys[project_name]:\n",
+    "        if extract_name == \"meta\":\n",
+    "            print(f\"=== {extract_name} PARAFRAME ===\")\n",
+    "            print(tree[project_name][extract_name])\n",
+    "            print()\n",
+    "        elif extract_name == \"drives\":\n",
+    "            print(f\"=== {extract_name} PARAFRAME ===\")\n",
+    "            print(tree[project_name][extract_name])\n",
+    "            print()\n",
+    "        else:\n",
+    "            extract_keys = list(tree[project_name][extract_name].keys())\n",
+    "            all_extract_keys[f\"{project_name}/{extract_name}\"] = extract_keys\n",
+    "            print(f\"=== {extract_name} KEYS ===\")\n",
+    "            for key in extract_keys:\n",
+    "                print(key)\n",
+    "            print()\n",
+    "    print()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "366eb8f2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "for project_name in all_project_keys:\n",
+    "    print(f\"=== {project_name} PROJECT ===\")\n",
+    "    print()\n",
+    "    if \"meta\" in tree[project_name]:\n",
+    "        print(\"=== meta PARAFRAME ===\")\n",
+    "        print(tree[project_name][\"meta\"])\n",
+    "        print()\n",
+    "    elif \"drives\" in tree[project_name]:\n",
+    "        print(\"=== drives PARAFRAME ===\")\n",
+    "        print(tree[project_name][\"drives\"])\n",
+    "        print()\n",
+    "    for extract_name in all_project_keys[project_name]:\n",
+    "        extract_dict = tree[project_name][extract_name]\n",
+    "        if \"data\" in extract_dict:\n",
+    "            print(f\"=== {extract_name} FMTS ===\")\n",
+    "            for fmt in extract_dict[\"data\"]:\n",
+    "                print(f\"{fmt}\")\n",
+    "            print()\n",
+    "    print()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "557196c6",
+   "metadata": {},
+   "source": [
+    "inconsistant naming conventions makes it difficult to fair all files to the correct fmt"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e810a4f4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "for project_name in all_project_keys:\n",
+    "    print(f\"=== {project_name} PROJECT ===\")\n",
+    "    print()\n",
+    "    for extract_name in all_project_keys[project_name]:\n",
+    "        if extract_name != \"meta\" and extract_name != \"drives\":\n",
+    "            print(f\"=== {extract_name} EXTRACT ===\")\n",
+    "            print()\n",
+    "        extract_dict = tree[project_name][extract_name]\n",
+    "        if \"data\" in extract_dict:\n",
+    "            for fmt_str, stems in extract_dict[\"data\"].items():\n",
+    "                print(f\"=== {fmt_str} FMT PARAFRAME ===\")\n",
+    "                print(stems[\"all\"])\n",
+    "                print()\n",
+    "    print()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "base",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.13.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

doc/api.rst

@@ -0,0 +1,22 @@
+API Reference
+=============
+
+Repository State
+----------------
+# This tells Sphinx to import hallmark.filename and extract the module 
+# and function docstrings.
+
+.. automodule:: hallmark.repo
+   :members:
+
+.. automodule:: hallmark.repo_state
+   :members:
+
+.. automodule:: hallmark.state
+   :members:
+
+.. automodule:: hallmark.downloader
+   :members:
+
+.. automodule:: hallmark.paraframe
+   :members: