Skip to content

Bug: can't get portfolio deploy workflow working #2044

@tiffanychu90

Description

@tiffanychu90

Where did the bug occur?
Select from the below, and be sure to affix the appropriate label to this issue (e.g. dataset, jupyterhub, metabase, analysis.calitp.org)

  • Data (the warehouse)
  • JupyterHub
  • Metabase
  • analysis.calitp.org
  • specifically, it's venv error

Describe the bug

  • I can run a test portfolio site (build, deploy), and I used _section_analyses_test as a test because it uses the calitp.magics %%capture_parameters stuff.
  • But, when I put my portfolio, with just 2 entries, I get errors.
  • The errors show up on both the 2026.3.18 and 2026.2.6 images
  • The build and deploy are successful when I use the 2026.2.6 image + rewind portfolio/requirements.txt back to what it was, but since the index now is a GH action (this is not broken, but I think it makes the switch to multiple repos more difficult. Is the site managed by Terraform?)

To Reproduce

  • Getting the error to reproduce isn't super straightforward, as the error traces back to venv, and I'm not sure where it's coming from. I got a set of different errors.

Error 1

Never seen this - seems to be coming from whatever version typer was.
I have tried the portfolio deploy 3x in the last 2 weeks, and typer went from typer~=0.9 to typer=0.24 with uv sync. I think we're trying to not lock down specific versions, though having the upper bound for versions might be something to consider. How do we know what the upper bound is without running into errors / who will debug this and resolve for everyone?

TypeError: Secondary flag is not valid for non-boolean flag.

│ ❱ 631 │   app()                                                                                  │
│   632                                                                                            │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │                          app = <typer.main.Typer object at 0x78e974f32f90>                   │ │
│ │                DEPLOY_OPTION = <typer.models.OptionInfo object at 0x78e97c34f810>            │ │
│ │                         Dict = typing.Dict                                                   │ │
│ │                         enum = <module 'enum' from '/opt/conda/lib/python3.11/enum.py'>      │ │
│ │                          env = <jinja2.environment.Environment object at 0x78e97455bc90>     │ │
│ │      GOOGLE_ANALYTICS_TAG_ID = 'G-JCX3Z8JZJC'                                                │ │
│ │                     humanize = <module 'humanize' from                                       │ │
│ │                                '/opt/conda/lib/python3.11/site-packages/humanize/__init__.p… │ │
│ │                         json = <module 'json' from                                           │ │
│ │                                '/opt/conda/lib/python3.11/json/__init__.py'>                 │ │
│ │                         List = typing.List                                                   │ │
│ │                      logging = <module 'logging' from                                        │ │
│ │                                '/opt/conda/lib/python3.11/logging/__init__.py'>              │ │
│ │                     Optional = typing.Optional                                               │ │
│ │                           os = <module 'os' (frozen)>                                        │ │
│ │            papermill_engines = <papermill.engines.PapermillEngines object at 0x78e975691890> │ │
│ │                           pm = <module 'papermill' from                                      │ │
│ │                                '/opt/conda/lib/python3.11/site-packages/papermill/__init__.… │ │
│ │                PORTFOLIO_DIR = PosixPath('portfolio')                                        │ │
│ │ PORTFOLIO_PUBLISH_PRODUCTION = 'gs://calitp-analysis'                                        │ │
│ │    PORTFOLIO_PUBLISH_STAGING = 'gs://calitp-analysis-staging'                                │ │
│ │                           re = <module 're' from '/opt/conda/lib/python3.11/re/__init__.py'> │ │
│ │                    RESOLVERS = [<function district_name at 0x78e9749974c0>]                  │ │
│ │                       shutil = <module 'shutil' from '/opt/conda/lib/python3.11/shutil.py'>  │ │
│ │                    SITES_DIR = PosixPath('portfolio/sites')                                  │ │
│ │                   subprocess = <module 'subprocess' from                                     │ │
│ │                                '/opt/conda/lib/python3.11/subprocess.py'>                    │ │
│ │                          sys = <module 'sys' (built-in)>                                     │ │
│ │                         time = <module 'time' (built-in)>                                    │ │
│ │                        typer = <module 'typer' from                                          │ │
│ │                                '/opt/conda/lib/python3.11/site-packages/typer/__init__.py'>  │ │
│ │                         yaml = <module 'yaml' from                                           │ │
│ │                                '/opt/conda/lib/python3.11/site-packages/yaml/__init__.py'>   │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│ /opt/conda/lib/python3.11/site-packages/typer/main.py:328 in __call__                            │
│                                                                                                  │
│ /opt/conda/lib/python3.11/site-packages/typer/main.py:311 in __call__                            │
│                                                                                                  │
│ /opt/conda/lib/python3.11/site-packages/typer/main.py:350 in get_command                         │
│                                                                                                  │
│ /opt/conda/lib/python3.11/site-packages/typer/main.py:332 in get_group                           │
│                                                                                                  │
│ /opt/conda/lib/python3.11/site-packages/typer/main.py:483 in get_group_from_info                 │
│                                                                                                  │
│ /opt/conda/lib/python3.11/site-packages/typer/main.py:577 in get_command_from_info               │
│                                                                                                  │
│ /opt/conda/lib/python3.11/site-packages/typer/main.py:553 in                                     │
│ get_params_convertors_ctx_param_name_from_function                                               │
│                                                                                                  │
│ /opt/conda/lib/python3.11/site-packages/typer/main.py:877 in get_click_param                     │
│                                                                                                  │
│ /opt/conda/lib/python3.11/site-packages/typer/core.py:489 in __init__                            │
│                                                                                                  │
│ /opt/conda/lib/python3.11/site-packages/click/core.py:2883 in __init__                           │
╰──────────────────────────────────────────────────────────────────────────

Error 2: NoSuchKernel: pyproject_local_kernel_use_venv

NoSuchKernel: No such kernel named pyproject_local_kernel_use_venv, which is there on both images, with uv run python portfolio.py or python portfolio.py (older image). After getting the test _section_analyses_test to build and deploy on staging, I got this.

2026-05-12 16:12:49 - WARNING - papermill - Passed unknown parameter: name
Executing:   0%|                                                                           | 0/31 [00:00<?, ?cell/s]2026-05-12 16:12:49 - ERROR - traitlets - No such kernel named pyproject_local_kernel_use_venv
Executing:   0%|                                                                           | 0/31 [00:00<?, ?cell/s]
╭─────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────╮
│ /home/jovyan/data-analyses/portfolio/portfolio.py:491 in build                                                   │
│                                                                                                                  │
│   488 │   for part in portfolio_site.parts:                                                                      │
│   489 │   │   for chapter in part.chapters:                                                                      │
│   490 │   │   │   errors.extend(                                                                                 │
│ ❱ 491 │   │   │   │   chapter.generate(                                                                          │
│   492 │   │   │   │   │   execute_papermill=execute_papermill,                                                   │
│   493 │   │   │   │   │   continue_on_error=continue_on_error,                                                   │
│   494 │   │   │   │   │   prepare_only=prepare_only,                                                             │
│                                                                                                                  │
│ /home/jovyan/data-analyses/portfolio/portfolio.py:205 in generate                                                │
│                                                                                                                  │
│   202 │   │   │                                                                                                  │
│   203 │   │   │   if execute_papermill:                                                                          │
│   204 │   │   │   │   try:                                                                                       │
│ ❱ 205 │   │   │   │   │   pm.execute_notebook(                                                                   │
│   206 │   │   │   │   │   │   input_path=notebook,                                                               │
│   207 │   │   │   │   │   │   output_path=parameterized_path,                                                    │
│   208 │   │   │   │   │   │   parameters=self.resolved_params,                                                   │
│                                                                                                                  │
│ /home/jovyan/data-analyses/.venv/lib/python3.11/site-packages/papermill/execute.py:116 in execute_notebook       │
│                                                                                                                  │
│   113 │   │   │   kernel_name = papermill_engines.nb_kernel_name(engine_name=engine_name, nb=n                   │
│   114 │   │   │   # Execute the Notebook in `cwd` if it is set                                                   │
│   115 │   │   │   with chdir(cwd):                                                                               │
│ ❱ 116 │   │   │   │   nb = papermill_engines.execute_notebook_with_engine(                                       │
│   117 │   │   │   │   │   engine_name,                                                                           │
│   118 │   │   │   │   │   nb,                                                                                    │
│   119 │   │   │   │   │   input_path=input_path,                                                                 │
│                                                                                                                  │
│ /home/jovyan/data-analyses/.venv/lib/python3.11/site-packages/papermill/engines.py:49 in                         │
│ execute_notebook_with_engine                                                                                     │
│                                                                                                                  │
│    46 │                                                                                                          │
│    47 │   def execute_notebook_with_engine(self, engine_name, nb, kernel_name, **kwargs):                        │
│    48 │   │   """Fetch a named engine and execute the nb object against it."""                                   │
│ ❱  49 │   │   return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)                    │
│    50 │                                                                                                          │
│    51 │   def nb_kernel_name(self, engine_name, nb, name=None):                                                  │
│    52 │   │   """Fetch kernel name from the document by dropping-down into the provided engine                   │
│                                                                                                                  │
│ /home/jovyan/data-analyses/.venv/lib/python3.11/site-packages/papermill/engines.py:371 in execute_notebook       │
│                                                                                                                  │
│   368 │   │                                                                                                      │
│   369 │   │   nb_man.notebook_start()                                                                            │
│   370 │   │   try:                                                                                               │
│ ❱ 371 │   │   │   cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **k                   │
│   372 │   │   finally:                                                                                           │
│   373 │   │   │   nb_man.cleanup_pbar()                                                                          │
│   374 │   │   │   nb_man.notebook_complete()                                                                     │
│                                                                                                                  │
│ /home/jovyan/data-analyses/portfolio/portfolio.py:340 in execute_managed_notebook                                │
│                                                                                                                  │
│   337 │   @classmethod                                                                                           │
│   338 │   def execute_managed_notebook(cls, nb_man, kernel_name, **kwargs):                                      │
│   339 │   │   # call the papermill execution engine:                                                             │
│ ❱ 340 │   │   super().execute_managed_notebook(nb_man, kernel_name, **kwargs)                                    │
│   341 │   │                                                                                                      │
│   342 │   │   assert "original_parameters" in kwargs                                                             │
│   343                                                                                                            │
│                                                                                                                  │
│ /home/jovyan/data-analyses/.venv/lib/python3.11/site-packages/papermill/engines.py:443 in                        │
│ execute_managed_notebook                                                                                         │
│                                                                                                                  │
│   440 │   │   │   stdout_file=stdout_file,                                                                       │
│   441 │   │   │   stderr_file=stderr_file,                                                                       │
│   442 │   │   )                                                                                                  │
│ ❱ 443 │   │   return PapermillNotebookClient(nb_man, **final_kwargs).execute()                                   │
│   444                                                                                                            │
│   445                                                                                                            │
│   446 # Instantiate a PapermillEngines instance, register Handlers and entrypoints                               │
│                                                                                                                  │
│ /home/jovyan/data-analyses/.venv/lib/python3.11/site-packages/papermill/clientwrap.py:43 in execute              │
│                                                                                                                  │
│    40 │   │   if sys.version_info[0] == 3 and sys.version_info[1] >= 8 and sys.platform.starts                   │
│    41 │   │   │   asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())                        │
│    42 │   │                                                                                                      │
│ ❱  43 │   │   with self.setup_kernel(**kwargs):                                                                  │
│    44 │   │   │   self.log.info(f"Executing notebook with kernel: {self.kernel_name}")                           │
│    45 │   │   │   self.papermill_execute_cells()                                                                 │
│    46 │   │   │   info_msg = self.wait_for_reply(self.kc.kernel_info())                                          │
│                                                                                                                  │
│ /opt/conda/lib/python3.11/contextlib.py:137 in __enter__                                                         │
│                                                                                                                  │
│   134 │   │   # they are only needed for recreation, which is not possible anymore                               │
│   135 │   │   del self.args, self.kwds, self.func                                                                │
│   136 │   │   try:                                                                                               │
│ ❱ 137 │   │   │   return next(self.gen)                                                                          │
│   138 │   │   except StopIteration:                                                                              │
│   139 │   │   │   raise RuntimeError("generator didn't yield") from None                                         │
│   140                                                                                                            │
│                                                                                                                  │
│ /home/jovyan/data-analyses/.venv/lib/python3.11/site-packages/nbclient/client.py:600 in setup_kernel             │
│                                                                                                                  │
│    597 │   │   │   self.km = self.create_kernel_manager()                                                        │
│    598 │   │                                                                                                     │
│    599 │   │   if not self.km.has_kernel:                                                                        │
│ ❱  600 │   │   │   self.start_new_kernel(**kwargs)                                                               │
│    601 │   │                                                                                                     │
│    602 │   │   if self.kc is None:                                                                               │
│    603 │   │   │   self.start_new_kernel_client()                                                                │
│                                                                                                                  │
│ /home/jovyan/data-analyses/.venv/lib/python3.11/site-packages/jupyter_core/utils/__init__.py:165 in wrapped      │
│                                                                                                                  │
│   162 │   │   if not loop_running:                                                                               │
│   163 │   │   │   # No loop running, run the loop for this thread.                                               │
│   164 │   │   │   loop = ensure_event_loop()                                                                     │
│ ❱ 165 │   │   │   return loop.run_until_complete(inner)                                                          │
│   166 │   │                                                                                                      │
│   167 │   │   # Loop is currently running in this thread,                                                        │
│   168 │   │   # use a task runner.                                                                               │
│                                                                                                                  │
│ /opt/conda/lib/python3.11/asyncio/base_events.py:654 in run_until_complete                                       │
│                                                                                                                  │
│    651 │   │   if not future.done():                                                                             │
│    652 │   │   │   raise RuntimeError('Event loop stopped before Future completed.')                             │
│    653 │   │                                                                                                     │
│ ❱  654 │   │   return future.result()                                                                            │
│    655 │                                                                                                         │
│    656 │   def stop(self):                                                                                       │
│    657 │   │   """Stop running the event loop.                                                                   │
│                                                                                                                  │
│ /home/jovyan/data-analyses/.venv/lib/python3.11/site-packages/nbclient/client.py:550 in async_start_new_kernel   │
│                                                                                                                  │
│    547 │   │   ):                                                                                                │
│    548 │   │   │   self.extra_arguments += [f"--HistoryManager.hist_file={self.ipython_hist_fil                  │
│    549 │   │                                                                                                     │
│ ❱  550 │   │   await ensure_async(self.km.start_kernel(extra_arguments=self.extra_arguments, **                  │
│    551 │                                                                                                         │
│    552 │   start_new_kernel = run_sync(async_start_new_kernel)                                                   │
│    553                                                                                                           │
│                                                                                                                  │
│ /home/jovyan/data-analyses/.venv/lib/python3.11/site-packages/jupyter_core/utils/__init__.py:214 in ensure_async │
│                                                                                                                  │
│   211 │   if inspect.isawaitable(obj):                                                                           │
│   212 │   │   obj = cast(Awaitable[T], obj)                                                                      │
│   213 │   │   try:                                                                                               │
│ ❱ 214 │   │   │   result = await obj                                                                             │
│   215 │   │   except RuntimeError as e:                                                                          │
│   216 │   │   │   if str(e) == "cannot reuse already awaited coroutine":                                         │
│   217 │   │   │   │   # obj is already the coroutine's result                                                    │
│                                                                                                                  │
│ /home/jovyan/data-analyses/.venv/lib/python3.11/site-packages/jupyter_client/manager.py:97 in wrapper            │
│                                                                                                                  │
│    94 │   │   except Exception as e:                                                                             │
│    95 │   │   │   self._ready.set_exception(e)                                                                   │
│    96 │   │   │   self.log.exception(self._ready.exception())                                                    │
│ ❱  97 │   │   │   raise e                                                                                        │
│    98 │                                                                                                          │
│    99 │   return t.cast(F, wrapper)                                                                              │
│   100                                                                                                            │
│                                                                                                                  │
│ /home/jovyan/data-analyses/.venv/lib/python3.11/site-packages/jupyter_client/manager.py:88 in wrapper            │
│                                                                                                                  │
│    85 │   │   │   self._ready = _get_future()                                                                    │
│    86 │   │   try:                                                                                               │
│    87 │   │   │   # call wrapped method, await, and set the result or exception.                                 │
│ ❱  88 │   │   │   out = await method(self, *args, **kwargs)                                                      │
│    89 │   │   │   # Add a small sleep to ensure tests can capture the state before done                          │
│    90 │   │   │   await asyncio.sleep(0.01)                                                                      │
│    91 │   │   │   if self.owns_kernel:                                                                           │
│                                                                                                                  │
│ /home/jovyan/data-analyses/.venv/lib/python3.11/site-packages/jupyter_client/manager.py:441 in                   │
│ _async_start_kernel                                                                                              │
│                                                                                                                  │
│   438 │   │   │    and launching the kernel (e.g. Popen kwargs).                                                 │
│   439 │   │   """                                                                                                │
│   440 │   │   self._attempted_start = True                                                                       │
│ ❱ 441 │   │   kernel_cmd, kw = await self._async_pre_start_kernel(**kw)                                          │
│   442 │   │                                                                                                      │
│   443 │   │   # launch the kernel subprocess                                                                     │
│   444 │   │   self.log.debug("Starting kernel: %s", kernel_cmd)                                                  │
│                                                                                                                  │
│ /home/jovyan/data-analyses/.venv/lib/python3.11/site-packages/jupyter_client/manager.py:403 in                   │
│ _async_pre_start_kernel                                                                                          │
│                                                                                                                  │
│   400 │   │   if self.provisioner is None:  # will not be None on restarts                                       │
│   401 │   │   │   self.provisioner = KPF.instance(parent=self.parent).create_provisioner_insta                   │
│   402 │   │   │   │   self.kernel_id,                                                                            │
│ ❱ 403 │   │   │   │   self.kernel_spec,                                                                          │
│   404 │   │   │   │   parent=self,                                                                               │
│   405 │   │   │   )                                                                                              │
│   406 │   │   kw = await self.provisioner.pre_launch(**kw)                                                       │
│                                                                                                                  │
│ /home/jovyan/data-analyses/.venv/lib/python3.11/site-packages/jupyter_client/manager.py:196 in kernel_spec       │
│                                                                                                                  │
│   193 │   @property                                                                                              │
│   194 │   def kernel_spec(self) -> kernelspec.KernelSpec | None:                                                 │
│   195 │   │   if self._kernel_spec is None and self.kernel_name != "":                                           │
│ ❱ 196 │   │   │   self._kernel_spec = self.kernel_spec_manager.get_kernel_spec(self.kernel_nam                   │
│   197 │   │   return self._kernel_spec                                                                           │
│   198 │                                                                                                          │
│   199 │   cache_ports: Bool = Bool(                                                                              │
│                                                                                                                  │
│ /home/jovyan/data-analyses/.venv/lib/python3.11/site-packages/jupyter_client/kernelspec.py:287 in                │
│ get_kernel_spec                                                                                                  │
│                                                                                                                  │
│   284 │   │                                                                                                      │
│   285 │   │   resource_dir = self._find_spec_directory(kernel_name.lower())                                      │
│   286 │   │   if resource_dir is None:                                                                           │
│ ❱ 287 │   │   │   raise NoSuchKernel(kernel_name)                                                                │
│   288 │   │                                                                                                      │
│   289 │   │   return self._get_kernel_spec_by_name(kernel_name, resource_dir)                                    │
│   290                                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
NoSuchKernel: No such kernel named pyproject_local_kernel_use_venv

Expected behavior

  1. Portfolios can be built and deployed without analysts doing their own dependency management. We should pin down a working version. Test portfolio sites work, no problem, but somehow do not carry over into our actual portfolio sites. Why?

  2. Since I was able to get a version of the portfolio to deploy on the 2026.2.6 Image last week + was able to deploy portfolio a couple weeks back on the older image, I believe the changes were more recently introduced. Can we figure out which packages that need to be nailed down for the 2026.3.18 Image to work?

  3. GitHub action: we should be trigger the index to be rebuilt without a PR. The GH action looks for changes in portfolio/sites, but not every deploy would have such changes. We have 3 types of portfolio changes, the first 2 types are roughly split 50-50:

    • rerun an existing site, but portfolio/sites/my_site.yml table of contents are dynamically generated and can change with each deploy (any of the GTFS sites) - GH action would work as expected
    • rerun an existing site, but portfolio/sites/my_site.yml table of contents do not change, but contents of rendered notebooks do change (HQTA explorer, NTD sites where list in table of contents is stable, but contents inside notebooks is changing, or simply to re-deploy to make README changes) - GH action would not work as expected.
    • add new site or take down old site - GH action would work as expected

Metadata

Metadata

Assignees

Labels

adminAdministrative workbugSomething isn't workingportfolio workWork related to the analytics portfoliotoolingWork related to the management of our tooling and shared modules

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions