views of large systems take a long time

[michaeldeistler]

Together with @alexpejovic, we discovered that Views can be very slow, even to the point where it becomes a limiting factor. For example, in a network with many single-comp cells, the connect is very slow. See also the troubleshooting guide here.

This is because a View always tracks all elements of the Module, even if the user only needs some parts of it (e.g., .nodes). To fix this, we might consider a View to only build its attributes "on demand". For example, connect would only build the .nodes, but .vis() would only build .xyzr.

There might be other solutions, but I think we should consider the speed of Views in the future.

[jnsbck]

Might be worth considering as part of #646. I wonder if there is some attrs which cause the slowdown in particular or if its View in general.

We could make all the attrs of Module properties of View. This would only trigger the computation when the attribute is called. The only thing we'd need to update is nodes_in_view and edges_in_view essentially.

something like:

@property
def nodes(self):
    return self.base.nodes.loc[self._nodes_in_view]

[alexpejovic]

I did some testing and I found the cause of the issue, at least for the flyvis
network. When running cProfile, I can see that self.xyzr = self._xyzr_in_view()
in View __init__() takes almost two minutes for the flyvis network when running
pre_nodes = net.select(nodes=pre_nodes_array). However, I can't seem to reproduce
this result outside of the flyvis network. However, this fix works for me as of
now.

@@ -3799,7 +3799,6 @@ class View(Module):
         self._branchpoints = (
             ptr_nodes if ptr_nodes.empty else ptr_nodes.loc[self._branchpoints_in_view]
         )
-        self.xyzr = self._xyzr_in_view()
         self.ncomp = len(self.nodes)
         self.total_nbranches = len(self._branches_in_view)
         self.nbranches_per_cell = self._nbranches_per_cell_in_view()
@@ -3854,6 +3853,10 @@ class View(Module):
         if len(self.nodes) == 0:
             raise ValueError("Nothing in view. Check your indices.")

+    @functools.cached_property
+    def xyzr(self):
+        return self._xyzr_in_view()
+
     def _set_inds_in_view(
         self,
         pointer: Union[Module, View],

Should I make a PR to 1.0.0 for this change?

Also sorry for the insanely formatted diff, idk how to paste diffs into GitHub comments.