Error converting shapes due to mismatched size of X and s2

[noraharhen]

I'm getting this crazy error occasionally while running pybads with specify_target_noise=true. It looks like somehow s2 ends up with more elements than rows of X. Any idea what could be going on here?

Unfortunately, I cant create a reprex—it only happens occasionally with my own model fitting code...

Traceback (most recent call last):
  File "/dfs6/pub/nharhen/foraging/model_fitting/experiment1k_kids/cross_val/bads.py", line 39, in <module>
    result = bads.optimize()
  File "/data/homezvol3/nharhen/.local/lib/python3.9/site-packages/pybads/bads/bads.py", line 1239, in optimize
    ) = self._search_step_(gp)
  File "/data/homezvol3/nharhen/.local/lib/python3.9/site-packages/pybads/bads/bads.py", line 1554, in _search_step_
    gp, gp_exit_flag = local_gp_fitting(
  File "/data/homezvol3/nharhen/.local/lib/python3.9/site-packages/pybads/bads/gaussian_process_train.py", line 402, in local_gp_fitting
    gp, hyp_gp, res, exit_flag = _robust_gp_fit_(
  File "/data/homezvol3/nharhen/.local/lib/python3.9/site-packages/pybads/bads/gaussian_process_train.py", line 542, in _robust_gp_fit_
    new_hyp, _, res = tmp_gp.fit(
  File "/data/homezvol3/nharhen/.local/lib/python3.9/site-packages/gpyreg/gaussian_process.py", line 1008, in fit
    X, y, s2 = self._convert_shapes(X, y, s2)
  File "/data/homezvol3/nharhen/.local/lib/python3.9/site-packages/gpyreg/gaussian_process.py", line 2555, in _convert_shapes
    s2 = s2.reshape(N, 1)
ValueError: cannot reshape array of size 55 into shape (51,1)

[lacerbi]

Thanks for the report! This seems tough to debug if it happens randomly.
@GurjeetSinghSangra is looking into it but this might take a while.

[GurjeetSinghSangra]

Hello @noraharhen,
apologies for the late response, and thanks for reporting this.

I have been investigating this report. Unfortunately, by running Pybads on specified noisy tasks (specify_target_noise=true), I haven't been able to reproduce your problem, even for a high number of function evaluations (i.e. longer optimization runs) or high-dimension settings.

Moreover, by inspecting the most possible/likely parts of the code that should relate to your stack trace and related parts of the code. I do not find anything suspicious that could raise any errors or bugs.

I require a reproducible experiment to facilitate better debugging. The configuration setting should include your setting, which includes your function, prior bounds, and hopefully your random seed number.

[cgc]

I'm writing to add a reproducible example for this error developed by my colleague @JiaLinLi0822, based on a tutorial example. I'm using Python 3.12, numpy==2.3.3, and PyBADS==1.0.5.

import numpy as np
from pybads import BADS

np.random.seed(42343)

def noisy_sphere(x,sigma=1.0):
    """Simple quadratic function with added noise."""
    x_2d = np.atleast_2d(x)
    f = np.sum(x_2d**2, axis=1)
    noise = 1e-4 * sigma*np.random.normal(size=x_2d.shape[0])
    return f + noise

x0 = np.array([-3, -3]);      # Starting point
lower_bounds = np.array([-5, -5])
upper_bounds = np.array([5, 5])
plausible_lower_bounds = np.array([-2, -2])
plausible_upper_bounds = np.array([2, 2])

options = {
    "uncertainty_handling": True,
    "max_fun_evals": 300,
    "noise_final_samples": 100,
    "random_seed": 43234,
}

bads = BADS(noisy_sphere, x0, lower_bounds, upper_bounds, plausible_lower_bounds, plausible_upper_bounds, options=options)
optimize_result = bads.optimize()

This code results in the following error:

Traceback (most recent call last):
  File "/code/pybads/example.py", line 29, in <module>
    optimize_result = bads.optimize()
                      ^^^^^^^^^^^^^^^
  File "/code/pybads/env/lib/python3.12/site-packages/pybads/bads/bads.py", line 1243, in optimize
    ) = self._search_step_(gp)
        ^^^^^^^^^^^^^^^^^^^^^^
  File "/code/pybads/env/lib/python3.12/site-packages/pybads/bads/bads.py", line 1558, in _search_step_
    gp, gp_exit_flag = local_gp_fitting(
                       ^^^^^^^^^^^^^^^^^
  File "/code/pybads/env/lib/python3.12/site-packages/pybads/bads/gaussian_process_train.py", line 402, in local_gp_fitting
    gp, hyp_gp, res, exit_flag = _robust_gp_fit_(
                                 ^^^^^^^^^^^^^^^^
  File "/code/pybads/env/lib/python3.12/site-packages/pybads/bads/gaussian_process_train.py", line 542, in _robust_gp_fit_
    new_hyp, _, res = tmp_gp.fit(
                      ^^^^^^^^^^^
  File "/code/pybads/env/lib/python3.12/site-packages/gpyreg/gaussian_process.py", line 1008, in fit
    X, y, s2 = self._convert_shapes(X, y, s2)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/pybads/env/lib/python3.12/site-packages/gpyreg/gaussian_process.py", line 2555, in _convert_shapes
    s2 = s2.reshape(N, 1)
         ^^^^^^^^^^^^^^^^
ValueError: cannot reshape array of size 121 into shape (114,1)

Notably, this differs from the tutorial example by scaling down the noise term.

It looks like this error happens in the course of retrying GP fitting in _robust_gp_fit_. One strategy used to refit the GP is to drop data points (based on the option remove_points_after_tries). However, I suspect this code has a bug in it, because it mutates tmp_gp.s2 directly instead of s2, even though s2 will be passed into tmp_gp.fit(...) subsequently. Here's a patch that fixes this bug for me:

--- a/pybads/bads/gaussian_process_train.py
+++ b/pybads/bads/gaussian_process_train.py
@@ -572,8 +572,8 @@
                 X = X[~idx_drop_out]
                 Y = Y[~idx_drop_out]
                 # Remove also user specified noise
-                if tmp_gp.s2 is not None and tmp_gp.s2.size > 0:
-                    tmp_gp.s2 = tmp_gp.s2[~idx_drop_out]
+                if s2 is not None and s2.size > 0:
+                    s2 = s2[~idx_drop_out]
 
             # Retry with random sample prior
             old_hyp_gp = (

However, this generally makes me concerned that this section of code has been under-tested -- are there any existing unit tests that verify this remove_points_after_tries procedure works as anticipated?

[lacerbi]

Thanks for the very accurate report. @GurjeetSinghSangra has moved on to other projects and not sure if he would still have the time to look into this. Otherwise I will have a look, although it might take a bit. The provided snippets and analysis are very useful and should be enough for me to work on it.

[lacerbi]

@GurjeetSinghSangra worked on a patch thanks to the useful comments from @cgc and we think the problem is solved. He also added an explicit unit test for this. This was addressed in PR #57 and we subsequently made a new release.

If you update PyBADS with pip or conda this should be fixed. I'll close the issue but feel free to re-open if you still find problems, and thanks again for the very detailed and useful feedback!