Use leastRestrictive for mvappend element-type widening

[ahkcs]

Description

MVAppendFunctionImpl.updateMostGeneralType used strict Object.equals to compare each operand's component type against the running "most general" type, falling back to Calcite's ANY on any mismatch. Object.equals returns false for type pairs that differ only in nullability tag — e.g. array(1, 2) synthesizes INTEGER NULLABLE for its component while literal 3 is INTEGER NOT NULL — and for straightforwardly-widenable numerics like INT + DECIMAL.

The Calcite engine's enumerable runtime tolerates ANY because MVAppendImplementor.implement processes elements through Object[] — the declared element type is unused at execution time. The analytics-engine route is stricter: substrait can't serialize ANY, so isthmus throws UnsupportedOperationException: Unable to convert the type ANY during substrait conversion.

Two changes:

1. updateMostGeneralType widens via RelDataTypeFactory.leastRestrictive — the same routine SqlLibraryOperators.ARRAY uses for its return-type inference. For genuinely incompatible operand types (INT + VARCHAR, …) leastRestrictive returns null; fall back to ANY there to preserve the existing in-process Calcite engine Object[] runtime semantics that the mvappend(1, 'text', 2.5)-style tests rely on. Promote DECIMAL → DOUBLE on the way through: RowResponseCodec maps DECIMAL cells to FloatingPoint(DOUBLE) anyway, and an explicit DECIMAL element type triggers Calcite's element coercion to BigDecimal, which the JSON formatter renders inconsistently across paths.

2. MVAppendImplementor.implement pre-casts each scalar operand to the call's element Java class via EnumUtils.convert. Without this, Avatica's AbstractCursor.ArrayAccessor dispatches the per-element accessor by the declared SQL type — e.g. DoubleAccessor.getDouble does (Double) value — and would throw a runtime ClassCastException on an Integer cell when the call's element type widens to DOUBLE. Array operands pass through; their element-type alignment is the planner's responsibility.

A previous revision of this PR went straight to leastRestrictive widening and triggered a runtime corruption (mvappend(1, 2.5) → [0, 2.5]) for exactly the Avatica-cast reason above. The pre-cast in MVAppendImplementor resolves that without reintroducing the regression.

Test plan

• ./gradlew :core:test --tests "*MVAppend*" → green.
• ./gradlew :integ-test:integTest --tests "org.opensearch.sql.calcite.remote.CalciteMVAppendFunctionIT" (Calcite engine path) → 15/15 pass.
• ./gradlew :integ-test:integTest --tests "org.opensearch.sql.calcite.remote.CalciteArrayFunctionIT" → 60/60 pass (no regression on the sister IT).

Analytics-engine compatibility

Verified end-to-end against the analytics-engine path per the Mustang + SQL plugin SOP: cluster started with arrow-flight-rpc + opensearch-job-scheduler + analytics-engine + composite-engine + parquet-data-format + analytics-backend-lucene + analytics-backend-datafusion + opensearch-sql-plugin, this PR's SQL plugin republished to maven local on top of feature/ppl-coverage-bundle, the OpenSearch companion opensearch-project/OpenSearch#21554 applied (including its Register UDFs on FFM-created session contexts Rust fix), :integ-test:integTestRemote against the running cluster with tests.analytics.{parquet_indices,force_routing}=true.

CalciteMVAppendFunctionIT (force-routed through analytics-engine)

Status	Before	After
Passed	0/15 (was: 9/15 with conservative equalSansNullability-only widening)	10/15
Failed	15/15	5/15

Newly passing on the analytics-engine route as a result of this PR:

Test	Operand types	Why it now passes
testMvappendWithMixedArrayAndScalar	ARRAY<INT>, INT, INT	nullability-only mismatch → bridged by leastRestrictive
testMvappendWithComplexExpression	ARRAY<INT>, ARRAY<INT>, INT	nullability-only mismatch → bridged by leastRestrictive
testMvappendWithIntAndDouble	INT, DECIMAL	widens to DOUBLE via DECIMAL → DOUBLE promotion + operand pre-cast

The other 7 passing tests already passed under the prior equalSansNullability baseline (uniform-type mvappend signatures and pure-array operands).

Remaining 5 failures (all are documented architectural limits, not regressions)

#	Test	Operand types	Why it still fails
1	testMvappendWithMixedTypes	INT, VARCHAR, DECIMAL	genuinely heterogeneous → ARRAY<ANY> (substrait can't encode)
2	testMvappendWithFieldsAndLiterals	INT, VARCHAR, VARCHAR	INT + VARCHAR → ARRAY<ANY>
3	testMvappendWithEmptyArray	ARRAY<VARCHAR> (#5421 default), INT, INT	VARCHAR + INT → ARRAY<ANY>. The empty-array array() is bound to empty_arr via eval; at MVAppend's type-inference site we see the column reference, not the literal, so per-call detection can't reach back through the project chain.
4	testMvappendWithNull	VARCHAR, INT, INT	VARCHAR + INT → ARRAY<ANY> (nullif(1, 1) is INTEGER-typed, not NULL-typed)
5	testMvappendInWhereClause	VARCHAR, VARCHAR	analytics-engine OpenSearchFilterRule.resolveViableBackends extracts only the predicate's TOP operator (EQUALS) and checks it against the field's storage type (ARRAY); doesn't walk into nested calls, so array_length(combined) = 2 is rejected as "EQUALS on ARRAY field". Tracked separately as a planner refactor; not specific to mvappend.

Calcite's ANY is a JVM Object[] pass-through with no substrait/Arrow/DataFusion equivalent. Resolving (1)–(4) would require either changing PPL's mvappend contract from heterogeneous-Object[] to a uniform stringified type (breaks user expectations) or shipping Arrow Union arrays through the wire format (multi-PR upstream work). Documented in the corresponding "What's left" section of opensearch-project/OpenSearch#21554.

Companion changes

• Default empty array() return type to ARRAY<VARCHAR> #5421 (merged) — defaults empty array() element type to ARRAY<VARCHAR> in PPL's ArrayFunctionImpl. Required for empty-array tests in CalciteArrayFunctionIT (testMvjoinWithEmptyArray, testMvdedupWithEmptyArray, testMvzip*EmptyArray, testMvfindWithEmptyArray) to pass on the analytics-engine route.

• [Analytics Backend / DataFusion] Onboard PPL array constructor and 8 multivalue (mv) functions to analytics-engine route OpenSearch#21554 — onboards the eight PPL collection functions to the analytics-engine route, plus a load-bearing Rust-side fix that calls udf::register_all on FFM-created session contexts (without it, mvappend / mvfind / mvzip / convert_tz all fail end-to-end with "Unsupported function name" regardless of the SQL-side widening here).

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 6ea6ff7)

Here are some key observations to aid the review process:

🧪 No relevant tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ Recommended focus areas for review Possible Issue The boxedJavaClass method returns Object.class for unhandled SQL types (default case), but EnumUtils.convert at line 127 may fail or produce unexpected results when attempting to convert to Object.class. If elementClass is Object.class, the conversion is skipped (line 124), but if elementType is non-null with an unhandled SqlTypeName (e.g., DATE, TIMESTAMP, BINARY), the code will attempt conversion to Object.class, which may not behave as intended. private static Class<?> boxedJavaClass(SqlTypeName sqlType) { return switch (sqlType) { case BOOLEAN -> Boolean.class; case TINYINT -> Byte.class; case SMALLINT -> Short.class; case INTEGER -> Integer.class; case BIGINT -> Long.class; case FLOAT, REAL -> Float.class; case DOUBLE -> Double.class; case DECIMAL -> BigDecimal.class; case CHAR, VARCHAR -> String.class; default -> Object.class; }; } Type Safety When leastRestrictive returns null (line 97), the code falls back to ANY type but continues to attempt type coercion at line 127 with elementClass derived from ANY. The boxedJavaClass method returns Object.class for ANY, which causes the coercion to be skipped (line 124-125). However, this creates inconsistency: the element type is declared as ANY but individual elements retain their original types without uniform coercion, potentially causing runtime type mismatches in downstream code expecting homogeneous arrays. RelDataType least = typeFactory.leastRestrictive(java.util.List.of(current, candidate)); if (least == null) { return typeFactory.createSqlType(SqlTypeName.ANY); }

[github-actions]

PR Code Suggestions ✨

Latest suggestions up to 6ea6ff7

Explore these optional code suggestions:

Category	Suggestion	Impact
General	Enforce consistent type conversion logic The condition skips conversion when elementClass is Object.class, but this may lead to heterogeneous types in the result list. Verify that downstream code handles mixed types correctly, or consider enforcing type consistency even for Object.class to prevent runtime casting issues. core/src/main/java/org/opensearch/sql/expression/function/CollectionUDF/MVAppendFunctionImpl.java [124-128] -if (opType.getComponentType() != null || elementClass == Object.class) { +if (opType.getComponentType() != null) { coerced.add(op); } else { coerced.add(EnumUtils.convert(op, elementClass)); } Suggestion importance[1-10]: 4 __ Why: The suggestion questions the elementClass == Object.class condition, but the existing logic appears intentional: when the element type is Object, no conversion is needed since any type is acceptable. Removing this check could cause unnecessary conversions and potential issues.	Low
	Validate element type is non-null When elementType is null, the code defaults to Object.class, but this may mask underlying issues. Consider adding a null-check or logging to ensure elementType is expected to be null in valid scenarios, preventing silent failures. core/src/main/java/org/opensearch/sql/expression/function/CollectionUDF/MVAppendFunctionImpl.java [118-119] -Class<?> elementClass = - elementType == null ? Object.class : boxedJavaClass(elementType.getSqlTypeName()); +if (elementType == null) { + throw new IllegalStateException("Element type should not be null for array operations"); +} +Class<?> elementClass = boxedJavaClass(elementType.getSqlTypeName()); Suggestion importance[1-10]: 3 __ Why: While validation could be helpful, the code explicitly handles the null case by defaulting to Object.class, which appears intentional for handling edge cases. The suggestion to throw an exception would break this fallback behavior without clear evidence it's incorrect.	Low
	Use mutable list for compatibility Using java.util.List.of() creates an immutable list. If leastRestrictive internally modifies the list or expects a mutable collection, this could cause runtime failures. Consider using Arrays.asList() or explicitly creating a mutable ArrayList to avoid potential UnsupportedOperationException. core/src/main/java/org/opensearch/sql/expression/function/CollectionUDF/MVAppendFunctionImpl.java [96] -RelDataType least = typeFactory.leastRestrictive(java.util.List.of(current, candidate)); +RelDataType least = typeFactory.leastRestrictive(Arrays.asList(current, candidate)); Suggestion importance[1-10]: 2 __ Why: List.of() creates an immutable list, but leastRestrictive is a standard Calcite API method that only reads the list for type comparison. There's no evidence it modifies the list, making this suggestion overly cautious and unlikely to prevent actual issues.	Low

---

Previous suggestions

Suggestions up to commit 6cf2b65

Category	Suggestion	Impact
General	Add null safety for SQL type The elementType null check may be insufficient. If elementType.getSqlTypeName() returns null or an unmapped type, boxedJavaClass returns Object.class, but this could mask type resolution issues. Consider validating the SQL type name explicitly. core/src/main/java/org/opensearch/sql/expression/function/CollectionUDF/MVAppendFunctionImpl.java [118-119] -Class<?> elementClass = - elementType == null ? Object.class : boxedJavaClass(elementType.getSqlTypeName()); +SqlTypeName sqlTypeName = elementType == null ? null : elementType.getSqlTypeName(); +Class<?> elementClass = (elementType == null || sqlTypeName == null) + ? Object.class + : boxedJavaClass(sqlTypeName); Suggestion importance[1-10]: 5 __ Why: The suggestion adds defensive null checking for getSqlTypeName(), which could improve robustness. However, the boxedJavaClass method already handles unmapped types by returning Object.class in the default case, so this is a minor defensive improvement rather than fixing a critical bug.	Low
	Remove unnecessary fully qualified name Using the fully qualified java.util.List is unnecessary since List is already imported. This creates inconsistency with other usages in the file and reduces code readability. core/src/main/java/org/opensearch/sql/expression/function/CollectionUDF/MVAppendFunctionImpl.java [96] -RelDataType least = typeFactory.leastRestrictive(java.util.List.of(current, candidate)); +RelDataType least = typeFactory.leastRestrictive(List.of(current, candidate)); Suggestion importance[1-10]: 4 __ Why: The suggestion correctly identifies that java.util.List is redundant since List is already imported at line 12. However, this is a minor style improvement with minimal impact on functionality.	Low

Suggestions up to commit 5fc778f

Category	Suggestion	Impact
General	Use leastRestrictive for type widening Consider using SqlTypeUtil.leastRestrictive instead of manually checking nullability equality and creating nullable types. This aligns with the PR title and provides a more robust type widening mechanism that handles edge cases consistently with Calcite's type system. core/src/main/java/org/opensearch/sql/expression/function/CollectionUDF/MVAppendFunctionImpl.java [95-97] -if (SqlTypeUtil.equalSansNullability(typeFactory, current, candidate)) { - return typeFactory.createTypeWithNullability(current, true); +RelDataType leastRestrictive = SqlTypeUtil.leastRestrictive(typeFactory, List.of(current, candidate)); +if (leastRestrictive != null) { + return leastRestrictive; } Suggestion importance[1-10]: 4 __ Why: While SqlTypeUtil.leastRestrictive could provide more robust type widening, the PR's explicit comment explains that the current approach intentionally avoids widening genuinely different types (INT + DOUBLE, INT + VARCHAR) to preserve heterogeneous values. Using leastRestrictive might contradict this design decision by performing unwanted type coercion.	Low

Suggestions up to commit 5ab670b

Category	Suggestion	Impact
General	Add null check for candidate parameter Consider validating that current and candidate are non-null before calling leastRestrictive. While the method checks current == null earlier, candidate could theoretically be null, which might cause unexpected behavior in leastRestrictive. core/src/main/java/org/opensearch/sql/expression/function/CollectionUDF/MVAppendFunctionImpl.java [91-92] +if (candidate == null) { + return current; +} RelDataType least = typeFactory.leastRestrictive(java.util.List.of(current, candidate)); return least != null ? least : typeFactory.createSqlType(SqlTypeName.ANY); Suggestion importance[1-10]: 5 __ Why: The suggestion adds defensive null-checking for candidate, which could improve robustness. However, this is a minor improvement since the method likely receives non-null values from its callers in practice, and the score is moderate because it's primarily a defensive programming practice rather than fixing a critical bug.	Low

[github-actions]

Persistent review updated to latest commit 5fc778f

[github-actions]

Persistent review updated to latest commit 6cf2b65

[github-actions]

Persistent review updated to latest commit 6ea6ff7