Node Documentation

This node allows interacting with JavaScript-based alert, confirm, and prompt dialog boxes. When a WebBrowser shows a dialog, these can be dismissed or confirmed. Further, it allows to extract a dialog’s text and send keyboard input to the dialog. The node assumes, that a dialog box is currently shown in the browser window.

For this node to work with Chrome-based browsers or Firefox, you must set the unhandledPromptBehavior capability to ignore, otherwise the alerts will be automatically dismissed without the possibility of any interaction. You can use the predefined “Don’t auto-dismiss alerts” snippet in the Start WebDriver or WebDriver Factory node.

Important: This node cannot be used for HTTP authentication dialogs. In case you need to authenticate this way, you can send the credentials directly through the “Navigate” node. Add username and password to your URL like this: https://username:password@example.com (works with Chrome-based browsers and Firefox). Alternatively have a look at the Authenticate (HTTP Basic) node.

Configure authentication for HTTP Basic. Enter a username and password which will be used if a HTTP Basic authentication dialog would show up.

Note: This is only intended for HTTP Basic authentication – not for form-based authentication!

This node requires a Chrome-based browser.

Creates a factory loop for RemoteWebDrivers using different configurations.

Clears a WebElement in case it is a text entry element (input or textarea). In case the element is of different type, this node has no effect on the element. Note, that this does not fire any keyboard or mouse events. If you need this, use a Send Keys node instead, and send backspace keys to delete the element’s content, then send a tab key to ensure that a change event is fired.

Clicks on a WebElement. In case the click causes a new page to load, the execution of this action will attempt to block until the page has loaded (depending on various circumstances, it will not block, however). In this case, all existing WebElements referring to the previous page will become invalid, because the page is no longer available.

In case the click causes a new page to load via an event, the action will not block. In this case, you need to determine on your own, when the page has reloaded or simply wait for a fixed amount of time until you proceed with further execution.

“Prints” the current page content to a PDF. Works with Chromium-based browsers running in headless mode. The PDF file is appended as a “binary object” column. To write it to a file, you can e.g. use the “Binary Objects to Files” node.

Appends a new string column to the input table which contains the URL currently shown in the selected WebDriver.

Simulates drag and drop of a WebElement by a specified distance.

Executes JavaScript code in the current frame or window of the selected WebDriver. Data can be transferred between KNIME and the JavaScript code, by specifying a return type for the executed code and/or selecting input columns which provide argument values for the executed code.

Arguments can be accessed within JavaScript code through the arguments array. The following argument and return data types are supported:

The following snippet will return a JSON object which is mapped to a JSON cell in KNIME. For this, set the “Return type” to “JSON”:

const result = { property1: 'a', property2: 2, property3: [ 'b', 'c', 'c' ] };
return result;

In case you need to execute code asynchronously, enable the [asyncCallbackMethod] argument. In this case, the last argument given to the executed code is the callback which signals that the code has finished execution. In case you need to return results from your asynchronous code, provide it as argument to the callback method. For example, the following JavaScript code causes the browser to sleep for five seconds, then the execution continues:

const callback = arguments[arguments.length - 1]; // last argument is callback
window.setTimeout(function() {
  console.log('finished');
  callback(); // invoke the callback to continue execution
}, 5000);
        

Timeouts for asynchronous execution can be configured in the Start WebDriver node in the “Timeouts” tab: Script timeout.

This node extracts attribute values from a WebElement. For the specified attribute name, the attribute values are appended as string column to the input table. In case, an attribute is not present, a missing value cell is appended.

This node extracts a CSS property from a WebElement. For the specified CSS property name, the property values are appended as string column to the input table. In case, a CSS property is not present, a missing value cell is appended.

Note: “Shorthand” CSS properties such as background, font, border, etc. are usually not supported, instead you must use the explicit longhand property names, such as background-color, font-size, etc.

Color values are returned as RGBA strings, e.g. for a color property green Selenium will return rgba(0, 255, 0, 1).

This node appends a WebElement’s text content by extracting the innerHTML, outerHTML, or textContent property via JavaScript.

Extracts WebElements based on different criteria. The input can either be a WebDriver (representing a browser’s window), or a WebElement (a subpart of the window). The following search criteria are available:

You can use the “Outline” view to easily create selector queries from the current DOM tree: Find the element you want to extract and double click on the corresponding line. A matching selector expression will be automatically inserted into the “Query” field.

Alternatively, you can select elements directly in the browser (works for Chromium-based browsers Chrome, Chromium): Click the “Select” button beside the “Query” input, switch to your browser window, and select any element on the page. A selector expression will be inserted into the “Query” field.

Please note that these feature require (1) a running browser instance which must show the actual data you want to extract, (2) the “full input data” to the node to be available.

To search for a text snippet within the DOM tree, use the search bar below the source.

For each extracted WebElement a new row is added to the output table.

Note that you cannot only search within the given input, but also navigate the entire DOM tree. For example, you can use the XPath query .. to find a WebElement’s parent.

Also note that you can only find on a DOM element level here, but not text content or attributes. To extract text, use this node to select the surrounding element, and then connect an “Extract InnerHTML” node to get the text. To extract attribute values, use an “Extract Attribute” node accordingly.

This node allows to select an HTML frame on a page. The frame can be located by different methods. After execution, the WebDriver operates within the selected frame. To go back to the parent frame afterwards, run this node with the “Back to parent frame” option.

When using “Name or ID” or “WebElement”, make sure to point directly to a <frame> or <iframe> element!

This node works similarly to the Start WebDriver node, except that it relies on a global pool of WebDrivers respective browsers. When the node is executed, it will try to retrieve an existing, idle WebDriver with the specified configuration from the pool, and only start a new one in case no WebDriver is available.

Instead of quitting the WebDriver (e.g. by resetting this node, or by running a Quit WebDriver node), it is handed back to the pool and can be reused later. WebDrivers managed by the pool will be quit simultaneously with KNIME.

The node performs a simple health check before handing out a pooled WebDriver. In case a WebDriver is not responding (e.g. because it has crashed, or it was killed by the user) it is eliminated from the pool.

Note, that the pooling mechanism is primarily intended for speeding up workflows which would start new browsers very frequently (e.g. in a busy loop). Pooled drivers keep their settings (cache, cookies, data, history, etc.), it is therefore not advisable to use the pooling mechanism when you require a “clean” browser, e.g. during testing.

Highlights WebElements by creating a border around them. The color and size of the border are taken from input row’s size and color properties (as specified by the Color Manager and Size Manager nodes).

Configures logging for Selenium and different drivers. This node is intended for debugging purposes, when reporting bugs, and not for productive usage.

Use the node as follows:

1. Place the node at the beginning of the workflow (we suggest to make use of the flow variable output port and connect it to the “Start WebDriver” node to ensure executing this node before starting a Selenium workflow).
2. Configure the logging granularity in the node configuration as desired.
3. The node outputs temporary paths for log files which can be accessed during execution of the workflow.
4. When saving the workflow, all logfiles will be copied into the workflow’s node internal directory.

Navigates the selected WebDriver. The navigation types “URL”, “Back”, “Forward”, and “Refresh” are supported.

Ends dumping network requests and write the results to a table. This requires a “Network Dump Start” node to be placed upstream in the workflow.

Note that once this node has executed and written the collected data, you can not re-run it. Instead you need to re-run the workflow from the corresponding “Network Dump Start” node.

Starts dumping network requests for the provided WebDriver – this means that all requests and responses including their headers and body will be recorded so that they can be inspected later. Use the matching “Network Dump End” node to stop the dumping and access the results. Typically you’ll want to put this node straight after the “Start WebDriver” node.

This node requires a Chrome-based browser.

Adds a column with the currently loaded page source (i.e. the DOM) of the selected WebDriver. In case, the page was modified by JavaScript after page load, there is no guarantee, that the returned source actually reflects the currently displayed page (this depends on the selected WebDriver, see individual WebDriver documentation for more information).

This node quits a Selenium subworkflow by closing the WebDriver which was initially opened by a Start WebDriver node. This means that the browser instance opened at the beginning of the workflow is closed by this node. The node checks all rows and columns of the provided table for WebDriver cells.

This node contains no execution logic. Instead it will simply reset itself when a workflow is loaded. This can be useful when placing it before a “Start WebDriver” node to ensure that the node always gets reset when opening a workflow. (therefore connect the red “Flow Variable” output port with a “Start WebDriver” node by dragging to the node’s left upper corner)

This node can be used free of charge, even without a valid Selenium Nodes license (see here).

Allows selecting or deselecting options within a menu which is represented by select tags. Selections can be made by different criterions: By numeric zero-index (e.g. select first element in a select), by an option’s value, by an option’s text, or simply selecting all options.

Note: In case you are trying to select multiple elements and the corresponding select construct does not have a multiple attribute, an error is thrown.

Appends two new string columns to the input table which contain URL of the Selenium server and the session ID for the given WebDriver.

This allows to make use of the WebDriver’s REST API.

Enter a file path into an input of type “file”. Note that when executing this node, it will not “send” the file itself, but rather enter the path to the file, similar to the “Send Keys” node. Uploading of the file typically happens, when submitting an associated form.

Simulates typing into a WebElement which accepts text. You can send special keys using belows’s code. Just copy the “representation” into the input. The “Null” key as used as termination for sending sequences; e.g. to send Ctrl+A, enter \uE009a\uE000.

Solves plain image Captchas (i.e. one which shows some more or less distorted, but human-readable text which is supposed to be entered by the user) using the 2Captcha Captcha solving API.

This node requires the captcha as PNG image cell – you can extract it using a combination of a “Find Elements” node (select the captcha image) and a “Take Screenshot” node to create an image cell of (only!) the captcha.

The node appends a column with the captcha solution and a flow variable imageCaptchaSolution – use this to fill out the input page on the website for the solution.

To solve reCAPTCHA and hCaptcha, have a look athe “Solve Captcha” node instead.

Solves Captchas from reCAPTCHA or hCaptcha using the 2Captcha Captcha solving API. The Captchas will be automatically detected on the page and then sent to the API for solving. Solving typically takes from a few seconds up to a minute. Once the solutions are available, they will be entered automatically into the DOM and the node finishes executing. From that point on you can consider the Captcha challenge to be sovled and continue with your usual operations.

Parts of this node is based on the excellent plugin collection puppeteer-extra by berstend̡̲̫̹̠̖͚͓̔̄̓̐̄͛̀͘.

This node only works with Chrome-based browsers.

This node starts a Selenium workflow by creating a new WebDriver. A WebDriver serves as an interface to a web browser, and allows to open web pages, control the browser, and extract data from it. The actual WebDriver to use must be configured using an upstream WebDriver Factory node.

The WebDriver is automatically closed, when this node is being reset, or the workflow or KNIME is closed. Additionally, you can explicitly close the browser with a Quit WebDriver node.

Important note: Because of the way Selenium works, KNIME workflows using the Selenium Nodes behave differently than you might be used from other nodes. As a Selenium workflow relies on a running browser instance, it is for example not possible to re-run a saved KNIME workflow from in between after it has been loaded from disk. In such cases, you need to re-run the workflow starting from the initial Start WebDriver node, as the downstream Selenium nodes require a WebDriver instance for operation.

Causes a form submit, in case the WebElement is either of type form, or it is contained within a form. If the submission causes a new page to load, the action will block until the page has loaded. In case the element the element is neither a form, nor contained in such, running this node will cause an error.

This node serves as a helper to synchronize a branched Selenium workflow. In case a Selenium workflow consists of parallel branches, and one wants to wait at a certain point until all branches have finished, all the branches can be connected to input ports of this node. Further execution blocks until all branches are available. The output port of this node provides the (unmodified) input of the first input port.

Note: There’s an improved successor of this node available as part of the free NodePit Power Nodes. Have a look at NoOp (Configurable).

This node conveniently converts HTML tables into KNIME tables. It uses some simple heuristics to determine the column names. The result are three KNIME tables. The first KNIME table contains the HTML table’s content, the second and third KNIME tables contain the HTML table’s header and footer.

As KNIME supports no spanning rows or columns, rowspan and colspan attributes in the HTML table are mapped by simply copying the original cell’s content to the spanning cells.

The HTML table’s header is detected by checking, whether (1) all cells are of type th, or (2) the cells are contained within a thead element. The HTML table’s footer is detected through the tfoot element.

The KNIME table’s column names are generated from HTML table’s header. In case the header is a single row, the column names equal the HTML table’s header names. In case there is more than one row in the HTML table’s header, the rows are concatenated with “ > ”. In case the HTML table contains no header, the column names are synthetically generated (“column0”, “column1”, …)

Here’s an example HTML table:

This will result in the following KNIME content table:

And the KNIME header table:

The footer table would be empty in this example, because the HTML table contains no footer.

Take a full page screenshot of the currently loaded page in a Chrome-based browser.

Take a screenshot of the currently loaded page in the selected WebDriver, or a specified WebElement. The WebDriver respective WebElement must support taking screenshots, else an error is triggered (e.g. the HtmlUnitDriver does not allow to take screenshot, as it is not running within a GUI). The extent of the resulting screenshot depends on the actual WebDriver (some WebDrivers capture the entire page, or the current window, or the visible frame, or the entire screen).

Quoting from the Selenium API documentation:

For WebDriver […], this makes a best effort depending on the browser to return the following in order of preference:

• Entire page
• Current window
• Visible portion of the current frame

For WebElement […], this makes a best effort depending on the browser to return the following in order of preference:

• The entire content of the HTML element
• The visible portion of the HTML element

This node simply pauses the execution for the specified amout of time. It can be used within Selenium workflows to wait for asynchronously loaded data.

This node pauses the execution until either the given conditions apply, or until the specified timeout has elapsed. If the condition does not apply within the given timeout, the node execution will either (a) fail completely, or (b) branch into the second output port, depending on the configuration.

To perform a simple check of the conditions without any wait, set the “Wait time” to zero.

The node creates the following flow variables:

• numMatched - the number of rows which matched the condition
• hasMatched - boolean flag to indicate that at least one row matched the condition
• numUnmatched - the number of rows which did not match the condition
• hasUnmatched - boolean flag to indicate that at least one row did not match the condition

These flow variables can be useful for controlling loop behavior (e.g. if you loop a paginated list and want to stop the loop once there’s no more pages). See “Variable Condition Loop End” node.

This node is responsible for configuring a WebDriver Factory. The WebDriver Factory creates new WebDrivers which are required for example by the Start WebDriver node. The node’s configuration allows to select a concrete WebDriver implementation. The following WebDrivers are supported by default:

• Chrome
• Edge
• Firefox
• HtmlUnit(deprecated) – HtmlUnit-based headless browser – it is the fastest and and most lightweight implementation of WebDriver. Pros: It runs in pure Java and is platform-independent. Cons: While it allows to run JavaScript if enabled, it uses its own engine “Rhino” which is different from other “real” browsers.
• Internet Explorer(not supported on this system)
• RemoteWebDriver – Allows to connect to a remote browser which is running on a server with Selenium Grid. You can also use this option to connect to third party, cloud-based Selenium services such as BrowserStack.
• Safari(not supported on this system)

All WebDrivers except HtmlUnit and the RemoteWebDriver start an actual browser on your machine. This means, that the browser needs to be installed and accessible on your system. Usually, the browsers can be found automatically, in case they are available in the system’s path variable or they are present at system-specific standard locations. See the WebDriver-specific documentations above for more information.

Capabilities for the WebDriver can be configured using the key-value editor. This is relevant when you need a to set specific browser settings or a proxy configuration. Chrome and Firefox can be launched “headless” — use the “Headless” capability snippets to enable.

This node allows to resize, maximize and position the WebDriver’s window. Beside that, the current dimensions can be retrieved.