Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/modules/automation-testing/nav.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -45,3 +45,7 @@

* Generate Appium test reports
** xref:automation-testing:generate-appium-test-reports/html-reports-for-appium-c-sharp.adoc[]

* Appium AI
** xref:appium-ai/overview-appium-ai.adoc[]
** xref:appium-ai/user-guide-appium-ai.adoc[]
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
= Appium AI: Natural language locators (Beta)

== Overview

Appium AI enables automation scripts to locate UI elements using natural language instead of traditional selectors such as XPath or accessibility IDs.

This allows test authors to describe elements in plain language while maintaining compatibility with existing Appium workflows.

This approach improves test resilience in environments where selectors are brittle, unavailable, or frequently changing.

== Availability

Appium AI (Natural Language Locators) is currently available in Beta.

This feature requires access to an OpenAI API or Azure OpenAI API. You must provide your own API credentials.

NOTE: Access to the OpenAI API is required and is separate from a ChatGPT subscription. ChatGPT access alone does not enable this feature.

This feature is available for private device deployments *only*. Appium AI is not available for Public Cloud devices.

Additional configuration may be required in deviceConnect. For setup assistance, contact Kobiton Support.

== How element location works in Appium

UI automation typically depends on selectors such as XPath, accessibility IDs, or view hierarchies to locate elements.

These selectors are often brittle and tightly coupled to the structure of the UI:

* Small UI changes can break existing tests
* Dynamic interfaces may not expose stable identifiers
* Some applications (such as canvas-based UIs) provide little or no metadata for automation

This increases maintenance effort and reduces the reliability of automated tests over time.

== Natural language locators

Appium AI extends existing Appium workflows by allowing `findElement(...)` to accept natural language descriptions instead of traditional selectors.

Instead of referencing elements by XPath or accessibility ID, you can describe the target element using plain language, such as:

----
python
element = driver.find_element("natural", "The login button at the bottom")
element.click()
----

or

----
java
driver.findElement(ByNatural.natural("the Accessibility button with an apostrophe"));
----

Appium AI interprets the description, identifies the most relevant UI element, and returns an interactable element for use in the test.

Natural language locators identify elements only. Actions such as `click()`, `send++_++keys()`, or assertions must still be performed explicitly in the test script.

This approach works within existing Appium scripts and does not require changes to client libraries or test structure.

NOTE: In Java, additional configuration may be required to enable the "natural" locator strategy in the Appium client.

== What this feature supports

Appium AI enables natural language-based element location for the following conditions:

* Using natural language descriptions with `findElement(...)` to locate UI elements
* Native and web application contexts where a view hierarchy is available
* Integration with existing Appium scripts without requiring structural changes

== Use cases

Use natural language locators in scenarios where traditional selectors are difficult to maintain or unreliable.

This approach is especially useful when:

* UI elements change frequently, causing selectors to break

* Applications use dynamic layouts or rendering patterns

* Readability and maintainability of test scripts are a priority

* Element identification is possible through visible labels or context

Natural language locators can help reduce maintenance overhead and improve test clarity in these situations.

== Limitations and considerations

Natural language locators are not intended to replace all selector strategies. Use them where they provide value, and rely on traditional selectors when precision is required.

This approach may not be suitable when:

* Stable and reliable selectors already exist

* Exact element matching is required (for example, when multiple similar elements are present)

* The application does not expose sufficient UI metadata

* The environment does not support natural language locators (such as Basic Appium or unsupported configurations)

Current limitations:

* Natural language locators identify elements only. They do not perform actions such as clicking or typing

* Script generation and full AI-driven test creation are not supported

* Canvas-based or fully visual UI element detection is not currently supported. Support for visual and canvas-based UI detection is planned for a future release.

* Automatic fallback from traditional locators to natural language is not supported

Natural language locators rely on available UI metadata (such as view hierarchy and accessibility attributes) to resolve elements. If this information is limited or unavailable, results may be less reliable.

== Next steps

To start using natural language locators in your tests, see the
xref:appium-ai/user-guide-appium-ai.adoc[Appium AI user guide] for setup instructions and examples.
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
= Use natural language locators in Appium AI

== Before you begin

Ensure the following:

* Your environment is configured to support natural language locators. If you are unsure, contact your lab administrator or Kobiton Support.
* Your test runs in a supported context (view hierarchy available)
* You are familiar with basic Appium commands such as `findElement(...)`
* You are using a private device deployment (natural language locators are not available for Public Cloud devices)

== Use a natural language locator

To locate an element using natural language, pass a descriptive string into `findElement(...)` using the "natural" locator strategy.

----
python
element = driver.find_element("natural", "The login button at the bottom")
element.click()
----

----
java
WebElement el = driver.findElement(ByNatural.natural("the Accessibility button with an apostrophe"));
el.click();
----

== Write effective descriptions

Use clear, specific descriptions to help identify the correct element.

Good examples:

----
java
driver.findElement(ByNatural.natural("the Accessibility button with an apostrophe"));
driver.findElement(ByNatural.natural("Enter email in the username field"));
driver.findElement(ByNatural.natural("Select Calgary from the city dropdown"));
----

Less effective examples:

----
driver.findElement(ByNatural.natural("Click button")
driver.findElement("Select item")
----

When multiple similar elements are present, include additional context such as position, label, or surrounding UI to improve accuracy.

== Expected behavior

Appium AI evaluates the description and returns the most relevant matching element based on available UI metadata.

Results are not guaranteed to be deterministic. If multiple elements match the description, you may need to refine the description to improve accuracy.

Natural language locators work alongside traditional selector strategies and are best used in combination with them, rather than as a complete replacement.

== Troubleshooting

If an element is not found or the wrong element is returned:

* Use more specific language in the description

* Include additional context such as position, labels, or surrounding elements

* Verify that the UI element is visible and accessible in the view hierarchy

* Confirm that the application exposes sufficient UI metadata for element identification

If natural language locators are not working as expected, contact your lab administrator to verify the following:

* OpenAI or Azure OpenAI API credentials are configured correctly

* Natural language locators are properly configured in your environment

* The deployment uses private devices (natural language locators are not available on Public Cloud devices)

If issues persist, contact Kobiton Support for assistance.

== Next steps

For an overview of how natural language locators work and their limitations, see the
xref:appium-ai/overview-appium-ai.adoc
[Appium AI] concept guide.
Loading