SOLR-8474: Add Selenium-based JUnit tests for the Admin UI - #4738
SOLR-8474: Add Selenium-based JUnit tests for the Admin UI#4738janhoy wants to merge 30 commits into
Conversation
- New opt-in JettyConfig.enableAdminUi flag makes JettySolrRunner serve the Admin UI static files and LoadAdminUiServlet like production web.xml does - New test sourceSet in solr/webapp with AdminUiTestBase: starts a 2-node cloud cluster with the UI enabled and drives it with headless Chrome via Selenium WebDriver; tests skip cleanly when no Chrome binary is found - First test: AdminUiDashboardTest asserts the dashboard displays versions, JVM and system stats matching the /admin/info/system API - Selenium 4.47.0 test-only dependency with license bookkeeping
Exclude opentelemetry (provided by solr-core), auto-service and jspecify annotation jars, and drop selenium-support in favor of a small poll-based wait helper. Only the 8 core selenium jars and a byte-buddy version bump remain as new dependencies.
Navigates every node-level, cloud, collection and core screen, waiting for each screen's main content element and asserting no severe browser console errors. Runs the test cluster with metricsEnabled=true so metrics-backed screens (Plugins) work.
Selenium requires 1.18.11 on the webapp test classpath; align the shared version so only a single byte-buddy version remains in solr/licenses.
Java properties, thread dump, logging tree, cloud nodes/tree views, collections detail, core admin, security and login screens, verified against the corresponding admin APIs. Serves a minimal stub of the generated js-client bundle so the Collections screen's CollectionsV2 service instantiates in tests.
Collection screens: query execution, analysis, schema browser, files, segments, plugins, documents form, paramsets, overview. Write actions through the UI: create and delete a collection, index a document, change and revert a log level - all verified via the corresponding APIs.
|
Well, this was impressive. In under one hour, Fable 5 did all this without any hand-holding beyond the initial promt (except me asking it to slim down dependencies). It figured out what tests to write, how to write them, how to wire in adminUI in test framework etc. The doc We'll then be able to catch regressions from any V2 work or any other commits to the UI. I ran all current 27 tests in 33 seconds, so this thing is fast too. Edit: I kicked off the next prompt, going warp speed here.. |
…ole log capture Adds createFixtureCollection/coreNameOnNode0/waitUntil helpers, an optional security.json for the test cluster, browser console logs in failure artifacts, a standard test log4j2 config so the log watcher sees events, and filters for two benign UI console errors.
…tions Groups the collections display and write tests in one feature class: create/delete collection, create/delete alias, add/delete replica and reload, each verified through the corresponding API.
Feature-grouped classes: query execution with rows/fl parameters, the documents indexing form, and paramset create/delete via the UI.
…gner Schema browser display, field flags and term info, add/delete field via the dialogs. Nightly Schema Designer happy path: new schema from a sample document.
Logging level editor and events viewer, core reload, streaming expression execution, and replication screen rendering in cloud mode.
… depth Removes AdminUiWriteActionsTest (its tests moved to the feature classes), trims the node/collection display classes accordingly, and deepens the cloud coverage: graph SVG replica circles and ZK status ensemble info.
Cluster bootstrapped with BasicAuth security.json; the login form flow, security screen display and adding a user via the dialog, verified via the authentication API.
…an doc Marks AdminUiSchemaDesignerTest @AwaitsFix (the designer backend is too flaky under automation), fixes the logging test's logger declaration, and adds a 'Possible UI bugs to investigate' section to the plan doc tracking the issues these tests surfaced, with the workarounds used.
Also lifts the class out of Nightly - it runs fast enough for the default suite - and moves the security.json setup from a static block to @BeforeClass, since test runners may load classes long before their suite executes.
The sql module jars are already licensed in solr/licenses, so this only adds lockfile entries. Also wires all Test-type tasks to the test sourceSet so the beast task works for this war project, and forwards the Chrome-binary override to all of them.
The cloud cluster now starts lazily on first use so subclasses can configure security.json or standalone mode in @BeforeClass; static-block configuration was order-dependent because test runners may load all classes up front. AdminUiStandaloneTestBase builds a no-ZooKeeper solr home and starts a standalone JettySolrRunner serving the UI. AdminUiCoreAdminStandaloneTest covers the standalone-only core admin actions - add, rename, swap (verified by the indexes exchanging) and unload (accepting the native confirm dialog) - plus the standalone menu differences. Adds a stale-safe setText helper for Angular-re-rendered form fields.
…lower pair Starts two standalone JettySolrRunners configured from the replication test configs; disables polling, indexes on the leader, replicates on demand via the UI button and re-enables polling - verified via the replication API.
The designer test stays AwaitsFix due to its backend's flakiness. The plan doc reflects the resolved TODOs (query depth, security dialogs, SQL, standalone core admin and replication) and the new findings, including the missing ui-grid icon font.
|
Now we have 53 tests and pretty wide test coverage. All tests ran in 47 secs on my M4 mac. I will call this the end of this experiment. Pretty successful if you ask me. |
There was a problem hiding this comment.
Pull request overview
Adds Selenium-driven JUnit coverage for the AngularJS Admin UI across cloud, standalone, replication, and authenticated configurations.
Changes:
- Adds opt-in Admin UI serving to
JettySolrRunner. - Introduces a reusable Chrome/Selenium harness and broad screen coverage.
- Adds Selenium dependencies, licenses, documentation, and lockfile updates.
Reviewed changes
Copilot reviewed 50 out of 50 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
solr/webapp/src/test/org/apache/solr/webapp/AdminUiTestBase.java |
Adds the shared browser and cloud test harness. |
solr/webapp/src/test/org/apache/solr/webapp/AdminUiStreamScreenTest.java |
Tests streaming expressions. |
solr/webapp/src/test/org/apache/solr/webapp/AdminUiStandaloneTestBase.java |
Adds standalone-node test support. |
solr/webapp/src/test/org/apache/solr/webapp/AdminUiSqlScreenTest.java |
Tests SQL form execution. |
solr/webapp/src/test/org/apache/solr/webapp/AdminUiSmokeTest.java |
Smoke-tests Admin UI routes. |
solr/webapp/src/test/org/apache/solr/webapp/AdminUiSecurityAuthTest.java |
Tests BasicAuth and security administration. |
solr/webapp/src/test/org/apache/solr/webapp/AdminUiSchemaScreenTest.java |
Tests schema browsing and field operations. |
solr/webapp/src/test/org/apache/solr/webapp/AdminUiSchemaDesignerTest.java |
Adds disabled Schema Designer coverage. |
solr/webapp/src/test/org/apache/solr/webapp/AdminUiReplicationStandaloneTest.java |
Tests standalone replication actions. |
solr/webapp/src/test/org/apache/solr/webapp/AdminUiReplicationScreenTest.java |
Tests cloud replication rendering. |
solr/webapp/src/test/org/apache/solr/webapp/AdminUiQueryScreenTest.java |
Tests query form parameters and execution. |
solr/webapp/src/test/org/apache/solr/webapp/AdminUiParamsetsScreenTest.java |
Tests parameter-set operations. |
solr/webapp/src/test/org/apache/solr/webapp/AdminUiNodeScreensTest.java |
Tests node and cloud screens. |
solr/webapp/src/test/org/apache/solr/webapp/AdminUiLoggingScreenTest.java |
Tests logging screens and level changes. |
solr/webapp/src/test/org/apache/solr/webapp/AdminUiDocumentsScreenTest.java |
Tests document indexing. |
solr/webapp/src/test/org/apache/solr/webapp/AdminUiDashboardTest.java |
Verifies dashboard system data. |
solr/webapp/src/test/org/apache/solr/webapp/AdminUiCoreAdminStandaloneTest.java |
Tests standalone core administration. |
solr/webapp/src/test/org/apache/solr/webapp/AdminUiCoreAdminScreenTest.java |
Tests cloud core administration. |
solr/webapp/src/test/org/apache/solr/webapp/AdminUiCollectionsScreenTest.java |
Tests collection and replica operations. |
solr/webapp/src/test/org/apache/solr/webapp/AdminUiCollectionScreensTest.java |
Tests collection and core detail screens. |
solr/webapp/src/test-files/log4j2.xml |
Configures test logging. |
solr/webapp/gradle.lockfile |
Locks browser-test dependencies. |
solr/webapp/build.gradle |
Configures Selenium tests and beasting. |
solr/test-framework/src/java/org/apache/solr/embedded/JettySolrRunner.java |
Serves Admin UI resources when enabled. |
solr/test-framework/src/java/org/apache/solr/embedded/JettyConfig.java |
Adds the Admin UI configuration option. |
solr/solrj/gradle.lockfile |
Updates Byte Buddy locking. |
solr/modules/s3-repository/gradle.lockfile |
Updates Byte Buddy locking. |
solr/modules/ltr/gradle.lockfile |
Updates Byte Buddy locking. |
solr/modules/jwt-auth/gradle.lockfile |
Updates Byte Buddy locking. |
solr/modules/cross-dc/gradle.lockfile |
Updates Byte Buddy locking. |
solr/modules/analysis-extras/gradle.lockfile |
Updates Byte Buddy locking. |
solr/licenses/selenium-remote-driver-4.47.0.jar.sha1 |
Adds the remote-driver checksum. |
solr/licenses/selenium-os-4.47.0.jar.sha1 |
Adds the Selenium OS checksum. |
solr/licenses/selenium-NOTICE.txt |
Adds Selenium attribution. |
solr/licenses/selenium-manager-4.47.0.jar.sha1 |
Adds the Selenium Manager checksum. |
solr/licenses/selenium-LICENSE-ASL.txt |
Adds the Selenium license. |
solr/licenses/selenium-json-4.47.0.jar.sha1 |
Adds the Selenium JSON checksum. |
solr/licenses/selenium-http-4.47.0.jar.sha1 |
Adds the Selenium HTTP checksum. |
solr/licenses/selenium-chromium-driver-4.47.0.jar.sha1 |
Adds the Chromium driver checksum. |
solr/licenses/selenium-chrome-driver-4.47.0.jar.sha1 |
Adds the Chrome driver checksum. |
solr/licenses/selenium-api-4.47.0.jar.sha1 |
Adds the Selenium API checksum. |
solr/licenses/byte-buddy-agent-1.18.9.jar.sha1 |
Removes the obsolete agent checksum. |
solr/licenses/byte-buddy-agent-1.18.11.jar.sha1 |
Adds the updated agent checksum. |
solr/licenses/byte-buddy-1.18.9.jar.sha1 |
Removes the obsolete Byte Buddy checksum. |
solr/licenses/byte-buddy-1.18.11.jar.sha1 |
Adds the updated Byte Buddy checksum. |
solr/cross-dc-manager/gradle.lockfile |
Updates Byte Buddy locking. |
solr/core/gradle.lockfile |
Updates Byte Buddy locking. |
gradle/libs.versions.toml |
Declares Selenium and updates Byte Buddy. |
dev-docs/admin-ui-tests.md |
Documents coverage, execution, and limitations. |
changelog/unreleased/admin-ui-selenium-tests.yml |
Records the new test suite. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Just ran in 34s on my M4 Air! Second time it burped on some tests, so looking at that. Pretty successful I ask you! |
|
This fails for me reproducibly. One thing is I don't know how useSecurityManager=true snuck in? |
janhoy
left a comment
There was a problem hiding this comment.
Only failing crave test is now the known TestGracefulJettyShutdown.
I have skimmed throgh the code, understand the architecture, how the UI is enabled in test framework, how selenium operates (used it before), and for all the individual tests I have NOT reviewed them all, but read through a few of them and I think they are easy to read and follow what is "clicked" etc. Since this touches test code and not produciton code I'm comfortable merging without a line-by-line review.
I'm happy with the stability of tests, first iteration had some multi jvm issues but beasting is now successful.
I left the tests enabled on a default test run, adding only ~ 1 minute test time. There should perhaps be a toggle for disabling UI tests short of adding test exclude pattern? If chrome cannot be found on the box, the tests are silently skipped, like we already do for certain other tests that require some deps, like TestContainers/docker.
epugh
left a comment
There was a problem hiding this comment.
I've played iwth the code, pushed up one fix that I found, and had it run locally for me succesfully.
My one question was in the admin-ui-test.md docs, and then I would change this to Approve!
| saved into the test temp dir. | ||
| - Run with: `./gradlew :solr:webapp:test` | ||
|
|
||
| ## Coverage by screen |
There was a problem hiding this comment.
this docs, while nice to see in hte PR< I don't think need to stay long term?
I wish you coul dlabel content "Here for hte life of the PR, but once merged to main it goes away"!
| the dialogs are driven via the Angular controller scope instead. Keyboard | ||
| entry is covered by the login form and the other screens' forms. | ||
|
|
||
| ## Possible UI bugs to investigate |
There was a problem hiding this comment.
and I guess we could have a JIRA for all of these seperate?
| `$scope.showCore` in `js/angular/app.js` — the core selector fires its | ||
| change handler with a null core while the menu resolves. Workaround: the | ||
| console-error assertion filters this signature. | ||
| 2. **Collections screen dies without the js-client bundle**: the |
There was a problem hiding this comment.
This specific bug doens't seem too important....
| metrics collection is disabled, instead of a clean error; the Plugins | ||
| screen just shows nothing while the console logs the 500. Workaround: | ||
| tests enable `metricsEnabled`. | ||
| 6. **Core overview ping widget logs a 503**: with a configset that has no |
There was a problem hiding this comment.
i think i saw a ticket about "add healthcheck to one of our default configsets" maybe because of this! yes, handle missing item.
| logs a 404 for it in production too, and grid icons render as boxes. | ||
| Workaround: the console-error assertion filters this 404. | ||
|
|
||
| ## Known limitations |
There was a problem hiding this comment.
some o fthese items might be resolved in the "migrate to v2" work, jthere were some weaknesses in events identified...
| See the License for the specific language governing permissions and | ||
| limitations under the License. | ||
| --> | ||
| <!-- We're configuring testing to be synchronous due to "logging polution", see SOLR-13268 --> |
There was a problem hiding this comment.
typo in the word polution....
| throws Exception { | ||
| ensureCloudCluster(); | ||
| cluster.uploadConfigSet(ExternalPaths.DEFAULT_CONFIGSET, name); | ||
| CollectionAdminRequest.Create create = |
There was a problem hiding this comment.
interesting, we have a SolrJ way of doing this, but I think we also generate a Java stub called CreateCollection that could be used here.
| saved into the test temp dir. | ||
| - Run with: `./gradlew :solr:webapp:test` | ||
|
|
||
| ## Coverage by screen |
There was a problem hiding this comment.
I don't. thinks this is useful... this just looks like a lot of content that isn't helpful to me, and will be out of date. Now, if we think AI needs this to understand the tests, but I don't think that either... Also, i wondered if this was a checked in "lets work our way through all the screens", but since everythign has a checked checkbox, i dont' think it's that either.
| `TypeError: Cannot read properties of null (reading 'name')` from | ||
| `$scope.showCore` while the per-collection menu resolves (filtered in the | ||
| console-error assertion; candidate for a JIRA). | ||
| - The core overview ping widget answers 503 when the configset has no |
There was a problem hiding this comment.
Should this item have a seperate JIRA to get fixed?
There was a problem hiding this comment.
(thanks for logging the other JS weaknesses as a JIRA)
https://issues.apache.org/jira/browse/SOLR-8474
The old (AngularJS) Admin UI has had no automated test coverage. This PR adds JUnit tests that drive the real UI in a headless Chrome browser via Selenium WebDriver — pure JVM dependencies, no Node.js toolchain.
What's included
JettyConfig.Builder#enableAdminUi(boolean)makesJettySolrRunner(and thusMiniSolrCloudCluster) serve the Admin UI static files andLoadAdminUiServletexactly as the productionweb.xmldoes — addressing the long-standingTODOinJettySolrRunner.solr/webapp/src/test):AdminUiTestBasestarts a 2-node cloud cluster with the UI enabled and a headless Chrome;AdminUiStandaloneTestBasecovers standalone (user-managed, no ZooKeeper) nodes — including a leader/follower pair — whose UI differs from cloud mode. Tests skip cleanly (JUnitAssume) on machines without Chrome — e.g. ASF Jenkins — and run wherever Chrome is installed (-Dtests.ui.chrome.binary=to override discovery). On failure a screenshot, the page source and the browser console log are saved. An optionalsecurity.jsonhook supports authenticated clusters, and a stub of the generated js-client bundle keeps the Collections screen alive from a source checkout.AdminUiSmokeTest— navigates every screen, asserting no severe console errorsAdminUiDashboardTest— versions/JVM/memory vs/admin/info/systemAdminUiNodeScreensTest— java properties, threads, cloud nodes/tree/graph/zkstatus, security, loginAdminUiCollectionsScreenTest— detail display; create/delete collection, create/delete alias, add/delete replica, reload — all verified via APIsAdminUiQueryScreenTest— query execution, rows/fl, paramsets dropdown, dismax/edismax toggles, raw query parametersAdminUiDocumentsScreenTest— indexing form; document indexed via UI becomes searchableAdminUiSchemaScreenTest— field browsing, flags and term info vs schema API; add/delete field via the dialogsAdminUiParamsetsScreenTest— create/delete paramset via the formAdminUiLoggingScreenTest— level editor set/unset; events viewerAdminUiCoreAdminScreenTest— core listing, reload core (cloud)AdminUiCoreAdminStandaloneTest— add/rename/swap/unload core on a standalone node, plus standalone menu differencesAdminUiCollectionScreensTest— analysis, files, segments, plugins, overviewsAdminUiStreamScreenTest— streaming expression executionAdminUiSqlScreenTest— SQL query via the form (sql module as a test-only dependency; its jars were already licensed)AdminUiReplicationScreenTest— cloud-mode renderingAdminUiReplicationStandaloneTest— leader/follower pair: disable polling, replicate-now, enable pollingAdminUiSecurityAuthTest— BasicAuth login flow, security screen, add user/role/permissionAdminUiSchemaDesignerTest(@AwaitsFix) — designer happy path; disabled until the designer's backend flakiness is addressedselenium-*jars (Apache-2.0) plus a byte-buddy version alignment; opentelemetry comes from solr-core,selenium-supportwas avoided by a small poll-based wait helper.gradlew -p solr/webapp beast -Ptests.dups=N --tests <Class>works (the generated test tasks needed explicit sourceSet wiring in this war project); the full suite is stable across order-shuffled beast runs.dev-docs/admin-ui-tests.mdThis PR was developed almost fully AI-assisted (Claude Fable).
Coverage checklist
@AwaitsFixdue to designer backend flakiness)Remaining TODOs for full coverage
ubuntu-latest)dev-docs/admin-ui-tests.md, and re-enableAdminUiSchemaDesignerTestonce the designer races are fixed