Skip to content

SOLR-8474: Add Selenium-based JUnit tests for the Admin UI - #4738

Open
janhoy wants to merge 30 commits into
apache:mainfrom
janhoy:admin-ui-selenium-tests
Open

SOLR-8474: Add Selenium-based JUnit tests for the Admin UI#4738
janhoy wants to merge 30 commits into
apache:mainfrom
janhoy:admin-ui-selenium-tests

Conversation

@janhoy

@janhoy janhoy commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

https://issues.apache.org/jira/browse/SOLR-8474

The old (AngularJS) Admin UI has had no automated test coverage. This PR adds JUnit tests that drive the real UI in a headless Chrome browser via Selenium WebDriver — pure JVM dependencies, no Node.js toolchain.

What's included

  • UI serving in tests: new opt-in JettyConfig.Builder#enableAdminUi(boolean) makes JettySolrRunner (and thus MiniSolrCloudCluster) serve the Admin UI static files and LoadAdminUiServlet exactly as the production web.xml does — addressing the long-standing TODO in JettySolrRunner.
  • Test harness (solr/webapp/src/test): AdminUiTestBase starts a 2-node cloud cluster with the UI enabled and a headless Chrome; AdminUiStandaloneTestBase covers standalone (user-managed, no ZooKeeper) nodes — including a leader/follower pair — whose UI differs from cloud mode. Tests skip cleanly (JUnit Assume) on machines without Chrome — e.g. ASF Jenkins — and run wherever Chrome is installed (-Dtests.ui.chrome.binary= to override discovery). On failure a screenshot, the page source and the browser console log are saved. An optional security.json hook supports authenticated clusters, and a stub of the generated js-client bundle keeps the Collections screen alive from a source checkout.
  • Feature-based test classes — each screen's display and write tests live together (53 tests in the default run):
    • AdminUiSmokeTest — navigates every screen, asserting no severe console errors
    • AdminUiDashboardTest — versions/JVM/memory vs /admin/info/system
    • AdminUiNodeScreensTest — java properties, threads, cloud nodes/tree/graph/zkstatus, security, login
    • AdminUiCollectionsScreenTest — detail display; create/delete collection, create/delete alias, add/delete replica, reload — all verified via APIs
    • AdminUiQueryScreenTest — query execution, rows/fl, paramsets dropdown, dismax/edismax toggles, raw query parameters
    • AdminUiDocumentsScreenTest — indexing form; document indexed via UI becomes searchable
    • AdminUiSchemaScreenTest — field browsing, flags and term info vs schema API; add/delete field via the dialogs
    • AdminUiParamsetsScreenTest — create/delete paramset via the form
    • AdminUiLoggingScreenTest — level editor set/unset; events viewer
    • AdminUiCoreAdminScreenTest — core listing, reload core (cloud)
    • AdminUiCoreAdminStandaloneTest — add/rename/swap/unload core on a standalone node, plus standalone menu differences
    • AdminUiCollectionScreensTest — analysis, files, segments, plugins, overviews
    • AdminUiStreamScreenTest — streaming expression execution
    • AdminUiSqlScreenTest — SQL query via the form (sql module as a test-only dependency; its jars were already licensed)
    • AdminUiReplicationScreenTest — cloud-mode rendering
    • AdminUiReplicationStandaloneTest — leader/follower pair: disable polling, replicate-now, enable polling
    • AdminUiSecurityAuthTest — BasicAuth login flow, security screen, add user/role/permission
    • AdminUiSchemaDesignerTest (@AwaitsFix) — designer happy path; disabled until the designer's backend flakiness is addressed
  • Dependencies kept slim: only the 8 core selenium-* jars (Apache-2.0) plus a byte-buddy version alignment; opentelemetry comes from solr-core, selenium-support was avoided by a small poll-based wait helper.
  • Beasting supported: gradlew -p solr/webapp beast -Ptests.dups=N --tests <Class> works (the generated test tasks needed explicit sourceSet wiring in this war project); the full suite is stable across order-shuffled beast runs.
  • Found real UI issues: the tests surfaced several UI bugs/weaknesses. They are tracked separately in SOLR-18347
  • Test catalog / status doc: dev-docs/admin-ui-tests.md

This PR was developed almost fully AI-assisted (Claude Fable).

Coverage checklist

  • Smoke navigation of every screen (node-level, cloud views, per-collection, per-core)
  • Dashboard display vs system API
  • Java properties, thread dump, logging tree displays
  • Cloud nodes/tree/graph/zkstatus displays
  • Collections: detail, create/delete, aliases, replicas, reload
  • Query: execution, rows/fl, paramsets dropdown, dismax/edismax toggles, raw query parameters
  • Documents: form + indexing via UI
  • Schema: browse, flags, term info, add/delete field
  • Paramsets: create/delete via UI
  • Logging: level change + events viewer
  • Core Admin: listing + reload (cloud); add/rename/swap/unload (standalone)
  • Analysis, Files, Segments, Plugins, Overviews
  • Stream expression execution
  • SQL screen (sql module on the test classpath)
  • Replication: cloud-mode rendering; standalone leader/follower replicate-now and polling toggles
  • Security with BasicAuth: login flow, security screen, add user/role/permission
  • Schema Designer happy path (@AwaitsFix due to designer backend flakiness)

Remaining TODOs for full coverage

  • JWT/OAuth login flows (need an identity provider — BasicAuth covers the login mechanics)
  • Serve the real js-client bundle when built, instead of the test stub
  • GitHub Actions workflow to run these tests in CI (Chrome is preinstalled on ubuntu-latest)
  • File JIRAs for the issues under "Possible UI bugs to investigate" in dev-docs/admin-ui-tests.md, and re-enable AdminUiSchemaDesignerTest once the designer races are fixed

janhoy added 9 commits August 13, 2026 23:25
- New opt-in JettyConfig.enableAdminUi flag makes JettySolrRunner serve the
  Admin UI static files and LoadAdminUiServlet like production web.xml does
- New test sourceSet in solr/webapp with AdminUiTestBase: starts a 2-node
  cloud cluster with the UI enabled and drives it with headless Chrome via
  Selenium WebDriver; tests skip cleanly when no Chrome binary is found
- First test: AdminUiDashboardTest asserts the dashboard displays versions,
  JVM and system stats matching the /admin/info/system API
- Selenium 4.47.0 test-only dependency with license bookkeeping
Exclude opentelemetry (provided by solr-core), auto-service and jspecify
annotation jars, and drop selenium-support in favor of a small poll-based
wait helper. Only the 8 core selenium jars and a byte-buddy version bump
remain as new dependencies.
Navigates every node-level, cloud, collection and core screen, waiting for
each screen's main content element and asserting no severe browser console
errors. Runs the test cluster with metricsEnabled=true so metrics-backed
screens (Plugins) work.
Selenium requires 1.18.11 on the webapp test classpath; align the shared
version so only a single byte-buddy version remains in solr/licenses.
Java properties, thread dump, logging tree, cloud nodes/tree views,
collections detail, core admin, security and login screens, verified
against the corresponding admin APIs. Serves a minimal stub of the
generated js-client bundle so the Collections screen's CollectionsV2
service instantiates in tests.
Collection screens: query execution, analysis, schema browser, files,
segments, plugins, documents form, paramsets, overview. Write actions
through the UI: create and delete a collection, index a document, change
and revert a log level - all verified via the corresponding APIs.
@janhoy

janhoy commented Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

Well, this was impressive. In under one hour, Fable 5 did all this without any hand-holding beyond the initial promt (except me asking it to slim down dependencies). It figured out what tests to write, how to write them, how to wire in adminUI in test framework etc. The doc dev-docs/admin-ui-tests.md has checkboxes for the remaining tests to write for full coverage. That is just one more prompt "implement the remaining tests we have planned", and then come back in an hour :)

We'll then be able to catch regressions from any V2 work or any other commits to the UI. I ran all current 27 tests in 33 seconds, so this thing is fast too.

Edit: I kicked off the next prompt, going warp speed here..

@janhoy
janhoy marked this pull request as draft August 13, 2026 22:20
@janhoy
janhoy requested a review from epugh August 13, 2026 22:20
janhoy added 7 commits August 14, 2026 00:27
…ole log capture

Adds createFixtureCollection/coreNameOnNode0/waitUntil helpers, an optional
security.json for the test cluster, browser console logs in failure
artifacts, a standard test log4j2 config so the log watcher sees events,
and filters for two benign UI console errors.
…tions

Groups the collections display and write tests in one feature class:
create/delete collection, create/delete alias, add/delete replica and
reload, each verified through the corresponding API.
Feature-grouped classes: query execution with rows/fl parameters, the
documents indexing form, and paramset create/delete via the UI.
…gner

Schema browser display, field flags and term info, add/delete field via
the dialogs. Nightly Schema Designer happy path: new schema from a sample
document.
Logging level editor and events viewer, core reload, streaming expression
execution, and replication screen rendering in cloud mode.
… depth

Removes AdminUiWriteActionsTest (its tests moved to the feature classes),
trims the node/collection display classes accordingly, and deepens the
cloud coverage: graph SVG replica circles and ZK status ensemble info.
janhoy added 9 commits August 14, 2026 00:55
Cluster bootstrapped with BasicAuth security.json; the login form flow,
security screen display and adding a user via the dialog, verified via
the authentication API.
…an doc

Marks AdminUiSchemaDesignerTest @AwaitsFix (the designer backend is too
flaky under automation), fixes the logging test's logger declaration, and
adds a 'Possible UI bugs to investigate' section to the plan doc tracking
the issues these tests surfaced, with the workarounds used.
Also lifts the class out of Nightly - it runs fast enough for the default
suite - and moves the security.json setup from a static block to
@BeforeClass, since test runners may load classes long before their suite
executes.
The sql module jars are already licensed in solr/licenses, so this only
adds lockfile entries. Also wires all Test-type tasks to the test
sourceSet so the beast task works for this war project, and forwards the
Chrome-binary override to all of them.
The cloud cluster now starts lazily on first use so subclasses can
configure security.json or standalone mode in @BeforeClass; static-block
configuration was order-dependent because test runners may load all
classes up front. AdminUiStandaloneTestBase builds a no-ZooKeeper solr
home and starts a standalone JettySolrRunner serving the UI.

AdminUiCoreAdminStandaloneTest covers the standalone-only core admin
actions - add, rename, swap (verified by the indexes exchanging) and
unload (accepting the native confirm dialog) - plus the standalone menu
differences. Adds a stale-safe setText helper for Angular-re-rendered
form fields.
…lower pair

Starts two standalone JettySolrRunners configured from the replication
test configs; disables polling, indexes on the leader, replicates on
demand via the UI button and re-enables polling - verified via the
replication API.
The designer test stays AwaitsFix due to its backend's flakiness. The
plan doc reflects the resolved TODOs (query depth, security dialogs, SQL,
standalone core admin and replication) and the new findings, including
the missing ui-grid icon font.
@janhoy

janhoy commented Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

Now we have 53 tests and pretty wide test coverage. All tests ran in 47 secs on my M4 mac. I will call this the end of this experiment. Pretty successful if you ask me.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds Selenium-driven JUnit coverage for the AngularJS Admin UI across cloud, standalone, replication, and authenticated configurations.

Changes:

  • Adds opt-in Admin UI serving to JettySolrRunner.
  • Introduces a reusable Chrome/Selenium harness and broad screen coverage.
  • Adds Selenium dependencies, licenses, documentation, and lockfile updates.

Reviewed changes

Copilot reviewed 50 out of 50 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
solr/webapp/src/test/org/apache/solr/webapp/AdminUiTestBase.java Adds the shared browser and cloud test harness.
solr/webapp/src/test/org/apache/solr/webapp/AdminUiStreamScreenTest.java Tests streaming expressions.
solr/webapp/src/test/org/apache/solr/webapp/AdminUiStandaloneTestBase.java Adds standalone-node test support.
solr/webapp/src/test/org/apache/solr/webapp/AdminUiSqlScreenTest.java Tests SQL form execution.
solr/webapp/src/test/org/apache/solr/webapp/AdminUiSmokeTest.java Smoke-tests Admin UI routes.
solr/webapp/src/test/org/apache/solr/webapp/AdminUiSecurityAuthTest.java Tests BasicAuth and security administration.
solr/webapp/src/test/org/apache/solr/webapp/AdminUiSchemaScreenTest.java Tests schema browsing and field operations.
solr/webapp/src/test/org/apache/solr/webapp/AdminUiSchemaDesignerTest.java Adds disabled Schema Designer coverage.
solr/webapp/src/test/org/apache/solr/webapp/AdminUiReplicationStandaloneTest.java Tests standalone replication actions.
solr/webapp/src/test/org/apache/solr/webapp/AdminUiReplicationScreenTest.java Tests cloud replication rendering.
solr/webapp/src/test/org/apache/solr/webapp/AdminUiQueryScreenTest.java Tests query form parameters and execution.
solr/webapp/src/test/org/apache/solr/webapp/AdminUiParamsetsScreenTest.java Tests parameter-set operations.
solr/webapp/src/test/org/apache/solr/webapp/AdminUiNodeScreensTest.java Tests node and cloud screens.
solr/webapp/src/test/org/apache/solr/webapp/AdminUiLoggingScreenTest.java Tests logging screens and level changes.
solr/webapp/src/test/org/apache/solr/webapp/AdminUiDocumentsScreenTest.java Tests document indexing.
solr/webapp/src/test/org/apache/solr/webapp/AdminUiDashboardTest.java Verifies dashboard system data.
solr/webapp/src/test/org/apache/solr/webapp/AdminUiCoreAdminStandaloneTest.java Tests standalone core administration.
solr/webapp/src/test/org/apache/solr/webapp/AdminUiCoreAdminScreenTest.java Tests cloud core administration.
solr/webapp/src/test/org/apache/solr/webapp/AdminUiCollectionsScreenTest.java Tests collection and replica operations.
solr/webapp/src/test/org/apache/solr/webapp/AdminUiCollectionScreensTest.java Tests collection and core detail screens.
solr/webapp/src/test-files/log4j2.xml Configures test logging.
solr/webapp/gradle.lockfile Locks browser-test dependencies.
solr/webapp/build.gradle Configures Selenium tests and beasting.
solr/test-framework/src/java/org/apache/solr/embedded/JettySolrRunner.java Serves Admin UI resources when enabled.
solr/test-framework/src/java/org/apache/solr/embedded/JettyConfig.java Adds the Admin UI configuration option.
solr/solrj/gradle.lockfile Updates Byte Buddy locking.
solr/modules/s3-repository/gradle.lockfile Updates Byte Buddy locking.
solr/modules/ltr/gradle.lockfile Updates Byte Buddy locking.
solr/modules/jwt-auth/gradle.lockfile Updates Byte Buddy locking.
solr/modules/cross-dc/gradle.lockfile Updates Byte Buddy locking.
solr/modules/analysis-extras/gradle.lockfile Updates Byte Buddy locking.
solr/licenses/selenium-remote-driver-4.47.0.jar.sha1 Adds the remote-driver checksum.
solr/licenses/selenium-os-4.47.0.jar.sha1 Adds the Selenium OS checksum.
solr/licenses/selenium-NOTICE.txt Adds Selenium attribution.
solr/licenses/selenium-manager-4.47.0.jar.sha1 Adds the Selenium Manager checksum.
solr/licenses/selenium-LICENSE-ASL.txt Adds the Selenium license.
solr/licenses/selenium-json-4.47.0.jar.sha1 Adds the Selenium JSON checksum.
solr/licenses/selenium-http-4.47.0.jar.sha1 Adds the Selenium HTTP checksum.
solr/licenses/selenium-chromium-driver-4.47.0.jar.sha1 Adds the Chromium driver checksum.
solr/licenses/selenium-chrome-driver-4.47.0.jar.sha1 Adds the Chrome driver checksum.
solr/licenses/selenium-api-4.47.0.jar.sha1 Adds the Selenium API checksum.
solr/licenses/byte-buddy-agent-1.18.9.jar.sha1 Removes the obsolete agent checksum.
solr/licenses/byte-buddy-agent-1.18.11.jar.sha1 Adds the updated agent checksum.
solr/licenses/byte-buddy-1.18.9.jar.sha1 Removes the obsolete Byte Buddy checksum.
solr/licenses/byte-buddy-1.18.11.jar.sha1 Adds the updated Byte Buddy checksum.
solr/cross-dc-manager/gradle.lockfile Updates Byte Buddy locking.
solr/core/gradle.lockfile Updates Byte Buddy locking.
gradle/libs.versions.toml Declares Selenium and updates Byte Buddy.
dev-docs/admin-ui-tests.md Documents coverage, execution, and limitations.
changelog/unreleased/admin-ui-selenium-tests.yml Records the new test suite.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread solr/webapp/src/test/org/apache/solr/webapp/AdminUiTestBase.java Outdated
@epugh

epugh commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Now we have 53 tests and pretty wide test coverage. All tests ran in 47 secs on my M4 mac. I will call this the end of this experiment. Pretty successful if you ask me.

Just ran in 34s on my M4 Air! Second time it burped on some tests, so looking at that. Pretty successful I ask you!

@epugh

epugh commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

This fails for me reproducibly. One thing is I don't know how useSecurityManager=true snuck in?

./gradlew :solr:webapp:test --tests "org.apache.solr.webapp.AdminUiReplicationStandaloneTest.testReplicationScreenAndActions" "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=BBFF98E2CBAD4116 -Ptests.useSecurityManager=true -Ptests.file.encoding=UTF-8

@janhoy
janhoy marked this pull request as ready for review August 14, 2026 08:10

@janhoy janhoy left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only failing crave test is now the known TestGracefulJettyShutdown.

I have skimmed throgh the code, understand the architecture, how the UI is enabled in test framework, how selenium operates (used it before), and for all the individual tests I have NOT reviewed them all, but read through a few of them and I think they are easy to read and follow what is "clicked" etc. Since this touches test code and not produciton code I'm comfortable merging without a line-by-line review.

I'm happy with the stability of tests, first iteration had some multi jvm issues but beasting is now successful.

I left the tests enabled on a default test run, adding only ~ 1 minute test time. There should perhaps be a toggle for disabling UI tests short of adding test exclude pattern? If chrome cannot be found on the box, the tests are silently skipped, like we already do for certain other tests that require some deps, like TestContainers/docker.

Comment thread dev-docs/admin-ui-tests.md Outdated
Comment thread solr/licenses/selenium-LICENSE-ASL.txt Outdated

@epugh epugh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've played iwth the code, pushed up one fix that I found, and had it run locally for me succesfully.

My one question was in the admin-ui-test.md docs, and then I would change this to Approve!

saved into the test temp dir.
- Run with: `./gradlew :solr:webapp:test`

## Coverage by screen

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this docs, while nice to see in hte PR< I don't think need to stay long term?

I wish you coul dlabel content "Here for hte life of the PR, but once merged to main it goes away"!

Comment thread dev-docs/admin-ui-tests.md Outdated
the dialogs are driven via the Angular controller scope instead. Keyboard
entry is covered by the login form and the other screens' forms.

## Possible UI bugs to investigate

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and I guess we could have a JIRA for all of these seperate?

Comment thread dev-docs/admin-ui-tests.md Outdated
`$scope.showCore` in `js/angular/app.js` — the core selector fires its
change handler with a null core while the menu resolves. Workaround: the
console-error assertion filters this signature.
2. **Collections screen dies without the js-client bundle**: the

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This specific bug doens't seem too important....

Comment thread dev-docs/admin-ui-tests.md Outdated
metrics collection is disabled, instead of a clean error; the Plugins
screen just shows nothing while the console logs the 500. Workaround:
tests enable `metricsEnabled`.
6. **Core overview ping widget logs a 503**: with a configset that has no

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think i saw a ticket about "add healthcheck to one of our default configsets" maybe because of this! yes, handle missing item.

logs a 404 for it in production too, and grid icons render as boxes.
Workaround: the console-error assertion filters this 404.

## Known limitations

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some o fthese items might be resolved in the "migrate to v2" work, jthere were some weaknesses in events identified...

See the License for the specific language governing permissions and
limitations under the License.
-->
<!-- We're configuring testing to be synchronous due to "logging polution", see SOLR-13268 -->

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo in the word polution....

throws Exception {
ensureCloudCluster();
cluster.uploadConfigSet(ExternalPaths.DEFAULT_CONFIGSET, name);
CollectionAdminRequest.Create create =

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

interesting, we have a SolrJ way of doing this, but I think we also generate a Java stub called CreateCollection that could be used here.

saved into the test temp dir.
- Run with: `./gradlew :solr:webapp:test`

## Coverage by screen

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't. thinks this is useful... this just looks like a lot of content that isn't helpful to me, and will be out of date. Now, if we think AI needs this to understand the tests, but I don't think that either... Also, i wondered if this was a checked in "lets work our way through all the screens", but since everythign has a checked checkbox, i dont' think it's that either.

`TypeError: Cannot read properties of null (reading 'name')` from
`$scope.showCore` while the per-collection menu resolves (filtered in the
console-error assertion; candidate for a JIRA).
- The core overview ping widget answers 503 when the configset has no

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this item have a seperate JIRA to get fixed?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(thanks for logging the other JS weaknesses as a JIRA)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants