diff --git a/docs/modules/automation-testing/nav.adoc b/docs/modules/automation-testing/nav.adoc index 3691eda7..bb40e416 100644 --- a/docs/modules/automation-testing/nav.adoc +++ b/docs/modules/automation-testing/nav.adoc @@ -45,3 +45,7 @@ * Generate Appium test reports ** xref:automation-testing:generate-appium-test-reports/html-reports-for-appium-c-sharp.adoc[] + +* Appium AI +** xref:appium-ai/overview-appium-ai.adoc[] +** xref:appium-ai/user-guide-appium-ai.adoc[] \ No newline at end of file diff --git a/docs/modules/automation-testing/pages/appium-ai/overview-appium-ai.adoc b/docs/modules/automation-testing/pages/appium-ai/overview-appium-ai.adoc new file mode 100644 index 00000000..21b65630 --- /dev/null +++ b/docs/modules/automation-testing/pages/appium-ai/overview-appium-ai.adoc @@ -0,0 +1,115 @@ += Appium AI: Natural language locators (Beta) + +== Overview + +Appium AI enables automation scripts to locate UI elements using natural language instead of traditional selectors such as XPath or accessibility IDs. + +This allows test authors to describe elements in plain language while maintaining compatibility with existing Appium workflows. + +This approach improves test resilience in environments where selectors are brittle, unavailable, or frequently changing. + +== Availability + +Appium AI (Natural Language Locators) is currently available in Beta. + +This feature requires access to an OpenAI API or Azure OpenAI API. You must provide your own API credentials. + +NOTE: Access to the OpenAI API is required and is separate from a ChatGPT subscription. ChatGPT access alone does not enable this feature. + +This feature is available for private device deployments *only*. Appium AI is not available for Public Cloud devices. + +Additional configuration may be required in deviceConnect. For setup assistance, contact Kobiton Support. + +== How element location works in Appium + +UI automation typically depends on selectors such as XPath, accessibility IDs, or view hierarchies to locate elements. + +These selectors are often brittle and tightly coupled to the structure of the UI: + +* Small UI changes can break existing tests +* Dynamic interfaces may not expose stable identifiers +* Some applications (such as canvas-based UIs) provide little or no metadata for automation + +This increases maintenance effort and reduces the reliability of automated tests over time. + +== Natural language locators + +Appium AI extends existing Appium workflows by allowing `findElement(...)` to accept natural language descriptions instead of traditional selectors. + +Instead of referencing elements by XPath or accessibility ID, you can describe the target element using plain language, such as: + +---- +python +element = driver.find_element("natural", "The login button at the bottom") +element.click() +---- + +or + +---- +java +driver.findElement(ByNatural.natural("the Accessibility button with an apostrophe")); +---- + +Appium AI interprets the description, identifies the most relevant UI element, and returns an interactable element for use in the test. + +Natural language locators identify elements only. Actions such as `click()`, `send++_++keys()`, or assertions must still be performed explicitly in the test script. + +This approach works within existing Appium scripts and does not require changes to client libraries or test structure. + +NOTE: In Java, additional configuration may be required to enable the "natural" locator strategy in the Appium client. + +== What this feature supports + +Appium AI enables natural language-based element location for the following conditions: + +* Using natural language descriptions with `findElement(...)` to locate UI elements +* Native and web application contexts where a view hierarchy is available +* Integration with existing Appium scripts without requiring structural changes + +== Use cases + +Use natural language locators in scenarios where traditional selectors are difficult to maintain or unreliable. + +This approach is especially useful when: + +* UI elements change frequently, causing selectors to break + +* Applications use dynamic layouts or rendering patterns + +* Readability and maintainability of test scripts are a priority + +* Element identification is possible through visible labels or context + +Natural language locators can help reduce maintenance overhead and improve test clarity in these situations. + +== Limitations and considerations + +Natural language locators are not intended to replace all selector strategies. Use them where they provide value, and rely on traditional selectors when precision is required. + +This approach may not be suitable when: + +* Stable and reliable selectors already exist + +* Exact element matching is required (for example, when multiple similar elements are present) + +* The application does not expose sufficient UI metadata + +* The environment does not support natural language locators (such as Basic Appium or unsupported configurations) + +Current limitations: + +* Natural language locators identify elements only. They do not perform actions such as clicking or typing + +* Script generation and full AI-driven test creation are not supported + +* Canvas-based or fully visual UI element detection is not currently supported. Support for visual and canvas-based UI detection is planned for a future release. + +* Automatic fallback from traditional locators to natural language is not supported + +Natural language locators rely on available UI metadata (such as view hierarchy and accessibility attributes) to resolve elements. If this information is limited or unavailable, results may be less reliable. + +== Next steps + +To start using natural language locators in your tests, see the +xref:appium-ai/user-guide-appium-ai.adoc[Appium AI user guide] for setup instructions and examples. diff --git a/docs/modules/automation-testing/pages/appium-ai/user-guide-appium-ai.adoc b/docs/modules/automation-testing/pages/appium-ai/user-guide-appium-ai.adoc new file mode 100644 index 00000000..ff7b4f28 --- /dev/null +++ b/docs/modules/automation-testing/pages/appium-ai/user-guide-appium-ai.adoc @@ -0,0 +1,84 @@ += Use natural language locators in Appium AI + +== Before you begin + +Ensure the following: + +* Your environment is configured to support natural language locators. If you are unsure, contact your lab administrator or Kobiton Support. +* Your test runs in a supported context (view hierarchy available) +* You are familiar with basic Appium commands such as `findElement(...)` +* You are using a private device deployment (natural language locators are not available for Public Cloud devices) + +== Use a natural language locator + +To locate an element using natural language, pass a descriptive string into `findElement(...)` using the "natural" locator strategy. + +---- +python +element = driver.find_element("natural", "The login button at the bottom") +element.click() +---- + +---- +java +WebElement el = driver.findElement(ByNatural.natural("the Accessibility button with an apostrophe")); +el.click(); +---- + +== Write effective descriptions + +Use clear, specific descriptions to help identify the correct element. + +Good examples: + +---- +java +driver.findElement(ByNatural.natural("the Accessibility button with an apostrophe")); +driver.findElement(ByNatural.natural("Enter email in the username field")); +driver.findElement(ByNatural.natural("Select Calgary from the city dropdown")); +---- + +Less effective examples: + +---- +driver.findElement(ByNatural.natural("Click button") +driver.findElement("Select item") +---- + +When multiple similar elements are present, include additional context such as position, label, or surrounding UI to improve accuracy. + +== Expected behavior + +Appium AI evaluates the description and returns the most relevant matching element based on available UI metadata. + +Results are not guaranteed to be deterministic. If multiple elements match the description, you may need to refine the description to improve accuracy. + +Natural language locators work alongside traditional selector strategies and are best used in combination with them, rather than as a complete replacement. + +== Troubleshooting + +If an element is not found or the wrong element is returned: + +* Use more specific language in the description + +* Include additional context such as position, labels, or surrounding elements + +* Verify that the UI element is visible and accessible in the view hierarchy + +* Confirm that the application exposes sufficient UI metadata for element identification + +If natural language locators are not working as expected, contact your lab administrator to verify the following: + +* OpenAI or Azure OpenAI API credentials are configured correctly + +* Natural language locators are properly configured in your environment + +* The deployment uses private devices (natural language locators are not available on Public Cloud devices) + +If issues persist, contact Kobiton Support for assistance. + +== Next steps + +For an overview of how natural language locators work and their limitations, see the +xref:appium-ai/overview-appium-ai.adoc +[Appium AI] concept guide. \ No newline at end of file