From fe28e07acf3fd46fbf1fe4fafe0e142afeaeabad Mon Sep 17 00:00:00 2001 From: tiffany-kobiton Date: Thu, 9 Apr 2026 15:08:04 -0400 Subject: [PATCH 1/3] Add Appium AI section and docs --- docs/modules/automation-testing/nav.adoc | 4 + .../pages/appium-ai/overview-appium-ai.adoc | 117 ++++++++++++++++++ .../pages/appium-ai/user-guide-appium-ai.adoc | 85 +++++++++++++ 3 files changed, 206 insertions(+) create mode 100644 docs/modules/automation-testing/pages/appium-ai/overview-appium-ai.adoc create mode 100644 docs/modules/automation-testing/pages/appium-ai/user-guide-appium-ai.adoc diff --git a/docs/modules/automation-testing/nav.adoc b/docs/modules/automation-testing/nav.adoc index 3691eda70..bb40e4167 100644 --- a/docs/modules/automation-testing/nav.adoc +++ b/docs/modules/automation-testing/nav.adoc @@ -45,3 +45,7 @@ * Generate Appium test reports ** xref:automation-testing:generate-appium-test-reports/html-reports-for-appium-c-sharp.adoc[] + +* Appium AI +** xref:appium-ai/overview-appium-ai.adoc[] +** xref:appium-ai/user-guide-appium-ai.adoc[] \ No newline at end of file diff --git a/docs/modules/automation-testing/pages/appium-ai/overview-appium-ai.adoc b/docs/modules/automation-testing/pages/appium-ai/overview-appium-ai.adoc new file mode 100644 index 000000000..695b78a95 --- /dev/null +++ b/docs/modules/automation-testing/pages/appium-ai/overview-appium-ai.adoc @@ -0,0 +1,117 @@ += Appium AI: Natural language locators (Beta) + +== Overview + +Appium AI enables automation scripts to locate UI elements using natural language instead of traditional selectors such as XPath or accessibility IDs. + +This allows test authors to describe elements in plain language while maintaining compatibility with existing Appium workflows. + +This approach improves test resilience in environments where selectors are brittle, unavailable, or frequently changing. + +== Availability + +Appium AI (Natural Language Locators) is currently available in Beta. + +This feature requires access to an OpenAI API or Azure OpenAI API. You must provide your own API credentials. + +NOTE: Access to the OpenAI API is required and is separate from a ChatGPT subscription. ChatGPT access alone does not enable this feature. + +This feature is available for private device deployments *only*. It is not supported in public cloud environments. + +Additional configuration may be required in deviceConnect. For setup assistance, contact Kobiton Support. + +== How element location works in Appium + +UI automation typically depends on selectors such as XPath, accessibility IDs, or view hierarchies to locate elements. + +These selectors are often brittle and tightly coupled to the structure of the UI: + +* Small UI changes can break existing tests +* Dynamic interfaces may not expose stable identifiers +* Some applications (such as canvas-based UIs) provide little or no metadata for automation + +This increases maintenance effort and reduces the reliability of automated tests over time. + +== Natural language locators + +Appium AI extends existing Appium workflows by allowing `findElement(...)` to accept natural language descriptions instead of traditional selectors. + +Instead of referencing elements by XPath or accessibility ID, you can describe the target element using plain language, such as: + +---- +python +element = driver.find_element("natural", "The login button at the bottom") +element.click() +---- + +or + +---- +java +driver.find_element("natural", "The login button at the bottom").click() +---- + +Appium AI interprets the description, identifies the most relevant UI element, and returns an interactable element for use in the test. + +Natural language locators identify elements only. Actions such as `click()`, `send++_++keys()`, or assertions must still be performed explicitly in the test script. + +This approach works within existing Appium scripts and does not require changes to client libraries or test structure. + +NOTE: In Java, additional configuration may be required to enable the "natural" locator strategy in the Appium client. + +== What this feature supports + +Appium AI enables natural language-based element location within existing Appium workflows. + +This includes: + +* Using natural language descriptions with `findElement(...)` to locate UI elements +* Native and web application contexts where a view hierarchy is available +* Integration with existing Appium scripts without requiring structural changes + +== Use cases + +Use natural language locators in scenarios where traditional selectors are difficult to maintain or unreliable. + +This approach is especially useful when: + +* UI elements change frequently, causing selectors to break + +* Applications use dynamic layouts or rendering patterns + +* Readability and maintainability of test scripts are a priority + +* Element identification is possible through visible labels or context + +Natural language locators can help reduce maintenance overhead and improve test clarity in these situations. + +== Limitations and considerations + +Natural language locators are not intended to replace all selector strategies. Use them where they provide value, and rely on traditional selectors when precision is required. + +This approach may not be suitable when: + +* Stable and reliable selectors already exist + +* Exact element matching is required (for example, when multiple similar elements are present) + +* The application does not expose sufficient UI metadata + +* The environment does not support natural language locators (such as Basic Appium or unsupported configurations) + +Current limitations: + +* Natural language locators identify elements only. They do not perform actions such as clicking or typing + +* Script generation and full AI-driven test creation are not supported + +* Canvas-based or fully visual UI element detection is not supported + +* Automatic fallback from traditional locators to natural language is not supported + +Natural language locators rely on available UI metadata (such as view hierarchy and accessibility attributes) to resolve elements. If this information is limited or unavailable, results may be less reliable. + +== Next steps + +To start using natural language locators in your tests, see the +xref:appium-ai/user-guide-appium-ai.adoc[Appium AI user guide] for setup instructions and examples. diff --git a/docs/modules/automation-testing/pages/appium-ai/user-guide-appium-ai.adoc b/docs/modules/automation-testing/pages/appium-ai/user-guide-appium-ai.adoc new file mode 100644 index 000000000..468dae0d6 --- /dev/null +++ b/docs/modules/automation-testing/pages/appium-ai/user-guide-appium-ai.adoc @@ -0,0 +1,85 @@ += Use natural language locators in Appium AI + +== Before you begin + +Ensure the following: + +* You are using an Appium-compatible environment that supports natural language locators +* Your test runs in a supported context (view hierarchy available) +* You are familiar with basic Appium commands such as `findElement(...)` +* You are using a private device deployment (natural language locators are not supported in public cloud environments) + +== Use a natural language locator + +To locate an element using natural language, pass a descriptive string into `findElement(...)` using the "natural" locator strategy. + +---- +python +element = driver.find_element("natural", "The login button at the bottom") +element.click() +---- + +---- +java +driver.find_element("natural", "The login button at the bottom").click() +---- + +== Write effective descriptions + +Use clear, specific descriptions to help identify the correct element. + +Good examples: + +---- +java +driver.findElement("Tap the login button").click() +driver.findElement("Enter email in the username field").click() +driver.findElement("Select Calgary from the city dropdown").click() +---- + +Less effective examples: + +---- +driver.findElement("Click button").click() +driver.findElement("Select item").click() +---- + +When multiple similar elements are present, include additional context such as position, label, or surrounding UI to improve accuracy. + +== Expected behavior + +Appium AI evaluates the description and returns the most relevant matching element based on available UI metadata. + +Results are not guaranteed to be deterministic. If multiple elements match the description, you may need to refine the description to improve accuracy. + +Natural language locators work alongside traditional selector strategies and are best used in combination with them, rather than as a complete replacement. + +== Troubleshooting + +If an element is not found or the wrong element is returned: + +* Use more specific language in the description + +* Include additional context such as position, labels, or surrounding elements + +* Verify that the UI element is visible and accessible in the view hierarchy + +* Confirm that the application exposes sufficient UI metadata for element identification + +If the feature is not working as expected: + +* Ensure your environment supports natural language locators + +* Confirm that your OpenAI or Azure OpenAI API credentials are configured correctly + +* Verify that the feature is enabled in your deviceConnect configuration (dc.ini) + +* Confirm that you are using a private device deployment (not supported in public cloud environments) + +If issues persist, contact Kobiton Support for assistance. + +== Next steps + +For an overview of how natural language locators work and their limitations, see the +xref:appium-ai/overview-appium-ai.adoc +[Appium AI] concept guide. \ No newline at end of file From 478654de541e85dc06de419d8110df0ebff4a132 Mon Sep 17 00:00:00 2001 From: tiffany-kobiton Date: Thu, 16 Apr 2026 09:39:50 -0400 Subject: [PATCH 2/3] Update Java syntax --- .../pages/appium-ai/overview-appium-ai.adoc | 2 +- .../pages/appium-ai/user-guide-appium-ai.adoc | 13 +++++++------ 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/modules/automation-testing/pages/appium-ai/overview-appium-ai.adoc b/docs/modules/automation-testing/pages/appium-ai/overview-appium-ai.adoc index 695b78a95..7fe27c6df 100644 --- a/docs/modules/automation-testing/pages/appium-ai/overview-appium-ai.adoc +++ b/docs/modules/automation-testing/pages/appium-ai/overview-appium-ai.adoc @@ -48,7 +48,7 @@ or ---- java -driver.find_element("natural", "The login button at the bottom").click() +driver.findElement(ByNatural.natural("the Accessibility button with an apostrophe")); ---- Appium AI interprets the description, identifies the most relevant UI element, and returns an interactable element for use in the test. diff --git a/docs/modules/automation-testing/pages/appium-ai/user-guide-appium-ai.adoc b/docs/modules/automation-testing/pages/appium-ai/user-guide-appium-ai.adoc index 468dae0d6..e96a38700 100644 --- a/docs/modules/automation-testing/pages/appium-ai/user-guide-appium-ai.adoc +++ b/docs/modules/automation-testing/pages/appium-ai/user-guide-appium-ai.adoc @@ -21,7 +21,8 @@ element.click() ---- java -driver.find_element("natural", "The login button at the bottom").click() +WebElement el = driver.findElement(ByNatural.natural("the Accessibility button with an apostrophe")); +el.click(); ---- == Write effective descriptions @@ -32,16 +33,16 @@ Good examples: ---- java -driver.findElement("Tap the login button").click() -driver.findElement("Enter email in the username field").click() -driver.findElement("Select Calgary from the city dropdown").click() +driver.findElement(ByNatural.natural("the Accessibility button with an apostrophe")); +driver.findElement(ByNatural.natural("Enter email in the username field")); +driver.findElement(ByNatural.natural("Select Calgary from the city dropdown")); ---- Less effective examples: ---- -driver.findElement("Click button").click() -driver.findElement("Select item").click() +driver.findElement(ByNatural.natural("Click button") +driver.findElement("Select item") ---- When multiple similar elements are present, include additional context such as position, label, or surrounding UI to improve accuracy. From 126e984bccd4191a7ab43a2dece3aa5217760c5a Mon Sep 17 00:00:00 2001 From: tiffany-kobiton Date: Wed, 6 May 2026 15:33:03 -0400 Subject: [PATCH 3/3] Update Appium AI overview and user guide per feedback --- .../pages/appium-ai/overview-appium-ai.adoc | 8 +++----- .../pages/appium-ai/user-guide-appium-ai.adoc | 14 ++++++-------- 2 files changed, 9 insertions(+), 13 deletions(-) diff --git a/docs/modules/automation-testing/pages/appium-ai/overview-appium-ai.adoc b/docs/modules/automation-testing/pages/appium-ai/overview-appium-ai.adoc index 7fe27c6df..21b65630a 100644 --- a/docs/modules/automation-testing/pages/appium-ai/overview-appium-ai.adoc +++ b/docs/modules/automation-testing/pages/appium-ai/overview-appium-ai.adoc @@ -16,7 +16,7 @@ This feature requires access to an OpenAI API or Azure OpenAI API. You must prov NOTE: Access to the OpenAI API is required and is separate from a ChatGPT subscription. ChatGPT access alone does not enable this feature. -This feature is available for private device deployments *only*. It is not supported in public cloud environments. +This feature is available for private device deployments *only*. Appium AI is not available for Public Cloud devices. Additional configuration may be required in deviceConnect. For setup assistance, contact Kobiton Support. @@ -61,9 +61,7 @@ NOTE: In Java, additional configuration may be required to enable the "natural" == What this feature supports -Appium AI enables natural language-based element location within existing Appium workflows. - -This includes: +Appium AI enables natural language-based element location for the following conditions: * Using natural language descriptions with `findElement(...)` to locate UI elements * Native and web application contexts where a view hierarchy is available @@ -105,7 +103,7 @@ Current limitations: * Script generation and full AI-driven test creation are not supported -* Canvas-based or fully visual UI element detection is not supported +* Canvas-based or fully visual UI element detection is not currently supported. Support for visual and canvas-based UI detection is planned for a future release. * Automatic fallback from traditional locators to natural language is not supported diff --git a/docs/modules/automation-testing/pages/appium-ai/user-guide-appium-ai.adoc b/docs/modules/automation-testing/pages/appium-ai/user-guide-appium-ai.adoc index e96a38700..ff7b4f289 100644 --- a/docs/modules/automation-testing/pages/appium-ai/user-guide-appium-ai.adoc +++ b/docs/modules/automation-testing/pages/appium-ai/user-guide-appium-ai.adoc @@ -4,10 +4,10 @@ Ensure the following: -* You are using an Appium-compatible environment that supports natural language locators +* Your environment is configured to support natural language locators. If you are unsure, contact your lab administrator or Kobiton Support. * Your test runs in a supported context (view hierarchy available) * You are familiar with basic Appium commands such as `findElement(...)` -* You are using a private device deployment (natural language locators are not supported in public cloud environments) +* You are using a private device deployment (natural language locators are not available for Public Cloud devices) == Use a natural language locator @@ -67,15 +67,13 @@ If an element is not found or the wrong element is returned: * Confirm that the application exposes sufficient UI metadata for element identification -If the feature is not working as expected: +If natural language locators are not working as expected, contact your lab administrator to verify the following: -* Ensure your environment supports natural language locators +* OpenAI or Azure OpenAI API credentials are configured correctly -* Confirm that your OpenAI or Azure OpenAI API credentials are configured correctly +* Natural language locators are properly configured in your environment -* Verify that the feature is enabled in your deviceConnect configuration (dc.ini) - -* Confirm that you are using a private device deployment (not supported in public cloud environments) +* The deployment uses private devices (natural language locators are not available on Public Cloud devices) If issues persist, contact Kobiton Support for assistance.