Fix jacobian and hessian evaluation for list expressions

[MrzvskK]

Motivation

Hi Johannes, when working on nonlinear constraints I have noticed that two tests were failing, my guess it is due to pandas.eval() and numexpr compatibility issues with list expressions in Python 3.13.
The tests in question:

• test_nonlinear_constraints_jacobian_expression
• test_nonlinear_constraints_hessian_expression

Cause analysis:
The jacobian() and hessian() methods were attempting to evaluate string expressions containing list literals such as "[2*x1, 2*x2, -1]" - directly with pandas.eval().
When doing so you get error message:
ValueError: setting an array element with a sequence due to inhomogeneous array shapes.

At first I created an issue, but then I made a few modifications to the nonlinear.py and now tests pass - I hope these fixes are good enough.
Both for Jacobian and Hessian we first detect list/nested list expressions, then these components are evaluated separately and we handle all the scalars by pushing them to all rows. Not sure if using dataFrame was a good call.

Changes

• jacobian(): Added parsing for list expressions like "[2*x1, 2*x2, -1]"
• hessian(): Added parsing for nested list expressions like "[[6*x1, 0, 0], [0, 2, 0], [0, 0, 0]]"

All other tests in the constraints also passed, and we maintain compatibility with Callable expressions and backward one for non-list expressions

Have you read the Contributing Guidelines on pull requests?

yes, but I am a new contributor

Have you updated CHANGELOG.md?

no

Test Plan

I would run: pytest tests/bofire/data_models/constraints/test_nonlinear.py -v
Or if you want to test all tests run: pytest tests/bofire/data_models/constraints/ -v

[TobyBoyne]

Hi all, just to clarify: pandas has two different eval engines. If numexpr is installed, pandas uses that backend which then causes the tests to fail. This is pretty surprising behaviour, that installing one extra package causes these tests to fail.

From speaking to @MrzvskK, her changes in this PR mean that the functions are now backend agnostic. The tests now pass independently of whether numexpr is installed.

[jduerholt]

Thanks for the explanations, I also digged a bit, so what I found is described in this two links:

• https://pandas.pydata.org/docs/reference/api/pandas.eval.html#pandas.eval
• https://pandas.pydata.org/docs/user_guide/enhancingperf.html

So if one does not set an engine explicitly via engine=python or engine=numexpr it will switch automatically to numexpr if installed and to python if numexpr is not available. From my point of view, we should decide fro one backend (and explicitly set the engine keyword) and then only support one backend. This makes the code cleaner and more readable.

What do you think and what are your preferences?

[TobyBoyne]

I agree that we should decide on a single backend. Since the pandas code isn't performance critical (all of the heavy lifting is done by torch or some optimization package like pymoo|SCIP|...), then we can just use the "python" backend since it seems to have the most lenient syntax?

Is there a way to globally set the pandas backend? Having to set backend="python" at every eval call seems a bit annoying but we should still do it.

[MrzvskK]

I do agree with the comments above. However, shouldn't we aim to have as much codebase as possible to be agnostic to these types of backend differences? I believe this strategy for Jacobian and Hessian computation wouldn't in fact have conflicts regardless of the version - also talking for future improvements.
But I let you decide, I don't know bofire that well to have a clear picture.

[jduerholt]

I would opt for now for the simple version, and just fix it everywhere to the python engine version. If then there is a need to go to something more speedy we can still migrate, especially as I am consedering to move at some point in the future to move to polars, as this is in general faster than pandas and has a less clunky api ...