Web Toolset Documentation

This toolset provides tools for making HTTP GET requests to retrieve web content.

Tool settings: host policies New in 0.9.14 Updated in 0.9.15

Both tools support host policies that control which servers they may be used to access. Configure them per tool under tool_settings::

• allowed_hosts: an array of host names that are always allowed without asking; when empty (default) no host is always allowed, so you will be asked before each access.
• denied_hosts: an array of host names to always refuse to connect to; when empty (default) no host is always denied.
• allow_private_hosts: set to true to allow access to private/local hosts; the default is false.

Behavior change As of 0.9.15 an unset allowed_hosts no longer means all hosts are allowed. A host that is neither allowed nor denied now triggers an interactive permission prompt (see below). With the approval prompts, the old allow-all-by-default (unsafe) fallback is gone — nothing is reachable without either a list entry or your explicit approval.

tool_settings:
  http_get_web_page_as_markdown:
    # always refused, no prompt
    denied_hosts: ["google.com"]
  http_get_web_page:
    # allowed without being asked
    allowed_hosts: ["google.com"]
    # any other host is neither allowed nor denied, so you'll be asked
    # each time before the request goes out

Permission prompts for unknown hosts New in 0.9.15

When a web tool fetches a host that appears in neither its allowed_hosts nor its denied_hosts list, Enkaidu asks for permission at each occurrence. In the terminal UI you get a four-letter choice:

• a — allow this one request
• d — deny this one request
• A — allow always
• D — deny always

Choosing allow-always or deny-always persists the host into that tool’s own allowed_hosts / denied_hosts list. Each tool keeps separate lists, so always-allowing a host for http_get_web_page_as_markdown has no effect on http_get_web_page (and vice versa). See Web UI security confirmations for the equivalent prompts in the web UI.

http_get_web_page

Makes an HTTP GET request to a given URL that returns text and returns the content type and content. It strips line indents and coalesces line breaks for HTML, JSON, and XML. Unless asked not to, it removes the head, script, and style elements as well as all class attributes from HTML content. Content size of response must be less than 32K.

Parameters

Notes

• Strips line indents and coalesces line breaks for HTML, JSON, and XML.
• Removes head, script, and style elements from HTML content unless preserve_source is set to true.
• Content size must be less than 32K.
• Follows redirects (301, 307, 308) up to a maximum of three times New in 0.9.16.

http_get_web_page_as_markdown

Makes an HTTP GET request to a given web site URL that returns the content as markdown, either because the website returns markdown directly, or by parsing the incoming HTML and converting it to markdown. If the text content from the web site is neither HTML nor markdown, the tool returns a markdown document with the web page content within a code block.

Parameters

Notes

• Returns content as markdown when the website supports it.
• Converts HTML to markdown by properly parsing the HTML.
• When the resulting content is too big (>32K), the markdown is truncated; the tool returns the URL plus a truncated: true | false property alongside the (possibly truncated) markdown New in 0.9.14.
• With save_as_file, the full page is written to the file — the >32K truncation does not apply to the saved file. This is the intended remedy for pages that would otherwise be truncated; the tool returns saved_as_file pointing at the written file New in 0.9.16.
• Content that is neither HTML nor markdown is wrapped in a code block with appropriate content type tagging where possible.
• Follows redirects (301, 307, 308) up to a maximum of three times New in 0.9.16.