Skip to main content

English Based Test Steps [Beta]

Overview​

English Based Test Steps brings natural language processing (NLP) using our DLM (Deterministic Language Model) to the Eyes SDK for Playwright (TS/JS Fixtures). Instead of writing selectors and interaction code, you describe each step in plain English and pass it to eyes.run(). The Applitools NLP engine parses the sentence, locates the target elements in the live page, and performs the actions in the order written.

Writing tests this way offers several advantages:

  • Readable and writable by anyone. A step like click the "Add to cart" button reads the same to a developer, a manual tester, or a product manager. Anyone on the team can understand what a test does, and can author or edit steps without Playwright expertise.
  • Resilient by design. Steps describe what the user sees, not how the DOM is built. The engine resolves elements at runtime from visible labels and attributes rather than fixed locators, so tests keep working through markup and selector changes that would break coded tests.
  • Private and secure. The engine is Applitools' own proprietary DLM, not an open-source or third-party LLM. Your application data stays safe with Applitools.
  • No token costs. NLP steps are included with your Applitools account at no additional charge and consume no LLM tokens or usage quota.
  • Integrated with Visual AI. NLP steps run inside the same test as eyes.check(), so you can interleave natural language actions with visual checkpoints and see both in the Playwright Fixtures report.

Prerequisites​

  • Supported in our Playwright TS/JS Fixtures SDK only
    • Requires @applitools/eyes-playwright version 1.49.0 or greater
    • Support for additional SDK frameworks is coming soon!
  • Node 24.16.0 or higher
  • Chrome, Chromium, Edge-Chromium browsers only (Firefox and WebKit are not supported)
  • Local or CI executions only; remote hosted test provider executions are not supported
  • Requires an active eyes.open session

Enabling NLP​

The nlpOptions.enabled setting is tri-state (true, false, or unset, which defers to your account entitlement). Set it at the playwright.config.ts project or global level.

// playwright.config.ts
eyesConfig: {
  appName: 'My App',
  nlpOptions: { enabled: true },
},

Note: Don't set nlpOptions inside test.use(), since it governs how the browser launches before any test runs.

Writing your first NLP step​

await eyes.run('type "John Doe" in the "Your Name" field and click the "Submit" button');

One sentence can hold several actions, executed in order.

Supported actions​

Click​

Click an element. Supports right click, double click, and modifier keys.

await eyes.run('click OK button');
await eyes.run('right-click the first "English" link');

Type​

Type text into an element.

await eyes.run('type "pants"');
await eyes.run('type secret "abcd" in password field');

Hover​

Move the mouse over an element.

await eyes.run('hover over "English"');

Focus​

Move keyboard focus to an element.

await eyes.run('Focus the OK button');

Check / Uncheck​

Check or uncheck a checkbox.

await eyes.run('tick the "delete" checkbox');
await eyes.run('uncheck the "dark mode" checkbox');

Clear​

Clear an element's value.

await eyes.run('clear search input');

Erase​

Delete a number of characters from an element.

await eyes.run('erase 5 characters on username input');

Set Value / Select​

Set a dropdown, date, time, or color picker value.

await eyes.run('Select the third option in the "Country" field');
await eyes.run('set date "10/13/2024" to the "Date Input"');

Hit Key / Press​

Press keyboard keys, including chords and repeats.

await eyes.run('hit enter');
await eyes.run('Press enter');

Go to a URL.

await eyes.run('Navigate to http://something.com/');

Go​

Move through browser history.

await eyes.run('Go back');
await eyes.run('Go back 2 pages');
await eyes.run('Go forward');

Scroll​

Scroll by distance or to an element.

await eyes.run('scroll down');
await eyes.run('Scroll to the "OK" button');

Refresh / Reload​

Reload the page, hard or soft.

await eyes.run('reload the page');
await eyes.run('Hard refresh the page');

Wait​

Pause for a duration, or until a condition is met.

await eyes.run('Wait 30 seconds for "Search" element to appear');

Alert​

Accept or close a native browser dialog.

await eyes.run('accept alert');
await eyes.run('close alert');

Close / Switch / Create Target​

Close, switch to, or open a browser tab.

await eyes.run('close browser tab');
await eyes.run('create browser tab');
await eyes.run('switch to the "Tab title" browser tab');

Table interaction​

Address a cell by visible content.

await eyes.run('Click the button at the 2nd row of the "Action" column');

Optionally​

Prefix any action to make it non-blocking if the target isn't found.

await eyes.run('Optionally click the "Accept Cookies"');

Verify and assertion syntax​

Assertions use assert, verify, or make sure, interchangeably, and support a wide range of comparisons.

await eyes.run('Verify that the Login button is visible');
await eyes.run('Assert the "Search" button is not visible');
await eyes.run('Verify that the price field starts with "$"');
await eyes.run('Verify that "quantity" equals 5');
await eyes.run('Verify that the "Email Address" field contains "@"');
await eyes.run('Verify that "Name" is "George Washington"');
await eyes.run('Assert that "Username" field is "joe smith" case insensitive');
await eyes.run('Verify that "Total Balance" is greater than $100');
await eyes.run('Verify that "Total Balance" is greater than $10 and less than $1000');
await eyes.run('Make sure that "Total Balance" is greater than "Available Balance"');
await eyes.run('Assert that the "Your name" field is capitalized');
await eyes.run('Validate that "good morning" appears');

A failing assertion throws EyesNlpError and will fail your test.

Non-supported actions​

The actions below are not available in this release. They may appear in other Applitools NLP documentation, such as Autonomous, but they are not currently supported through eyes.run() in this SDK.

Visual Check​

Performing a visual checkpoint from inside an NLP sentence is not yet supported.

await eyes.run('visual check'); // Not supported

Use eyes.check() directly instead, as shown throughout the examples on this page.

Generate​

Generating dynamic values (OTP codes, random test data) from within an NLP sentence is not supported.

await eyes.run('Generate a random first name of a female'); // Not supported

Set Value, file attachment form​

Selecting a dropdown, date, time, or color value is supported. Attaching a file is not.

await eyes.run('attach file "document.pdf" to "CV"'); // Not supported

Entering a query into a dedicated search element is not currently supported.

await eyes.run('search for "query" in searchbox'); // Not supported

Use type or enter to fill the search field, then click or press enter to submit it.

Close, in-page popups and dialogs​

Closing a native browser dialog or browser tab is supported. Closing an in-page popup, modal, or custom dialog element is not.

await eyes.run('dismiss the popup'); // Not supported

Set Variable​

Storing a value in a named variable for later use in a test is not currently supported.

await eyes.run('set var1 = "hello"'); // Not supported

Find​

Locating an element without interacting with it is an internal engine capability and is not exposed as a customer-facing action.

await eyes.run('find the "Login" button'); // Not supported

Evaluate​

Executing inline JavaScript from an NLP sentence is not supported.

await eyes.run('execute js: window.scrollBy(0, 10)'); // Not supported

Fetch​

Performing an HTTP request from an NLP sentence is not supported.

await eyes.run('get https://api-demo.com/api'); // Not supported

Grab​

Capturing a value into clipboard context for reuse in a later step is not supported.

await eyes.run('grab the page url and save it as "var1"'); // Not supported

Handling results and debugging​

eyes.run() returns an array with one result per executed action. The example below will return an array of three steps include their status and metadata.

const steps = await eyes.run('type "John Doe" in the "Your Name" field and "john.doe@acme.com" in the "Email" field and click the "Submit" button')
console.log(JSON.stringify(steps, null, 2))

Errors​

EyesNlpError is thrown on a failing step, naming the sentence and the reason for failure.

Logging and debugging​

Set APPLITOOLS_LOG_DIR and APPLITOOLS_SHOW_LOGS=1. Filter with grep '(nlp)' for step outcomes, or grep '(nlp/engine)' for the engine's own trace.

Example test​

The E-Commerce test below is a canonical end-to-end walkthrough, showing click, verify (including negated visibility), dropdown selection, optional actions, a multi-step statement, and visual checks together in one flow.

import { test } from '@applitools/eyes-playwright/fixture';
import assert from 'assert';

test('E-Commerce', async ({ page, eyes }) => {
  await page.goto('https://sandbox.applitools.com/e-commerce/');
  await eyes.check('landing page');

  await eyes.run("Click 'Small Succulent Planter Pot'");
  await eyes.run('Verify "Indescribable fungus nameless charnel." is visible');

  await eyes.run('Select "Gold" from the "Select Color" dropdown menu');
  await eyes.run("Click 'Add to cart' button");

  await eyes.run("Click the 'Contact Us' link");
  await eyes.run("Optionally click the 'Submit' button");
  await eyes.check('Contact Us');

  // Multi-step NLP statement
  const steps = await eyes.run(
    'type "John Doe" in the "Your Name" field and "john.doe@acme.com" in the "Email" field and select "Technical Assistance" from the "Contact Reason" dropdown menu and click the "Submit" button'
  );

  await eyes.check('Submitted contact form');
  console.log(JSON.stringify(steps, null, 2));

  await eyes.run("Click the 'shopping cart' button");
  await eyes.run("Verify 'Your cart is empty' is not visible");
});