p5-WWW-Robot-0.026p0 – configurable web traversal engine (cvsweb github mirror)

Description

This module implements a configurable web traversal engine, for a robot
or other web agent. Given an initial web page (URL), the Robot will get
the contents of that page, and extract all links on the page, adding
them to a list of URLs to visit.

Features of the Robot module include:

    * Follows the Robot Exclusion Protocol.
    * Supports the META element proposed extensions to the Protocol.
    * Implements many of the Guidelines for Robot Writers.
    * Configurable.
    * Builds on standard Perl 5 modules for WWW, HTTP, HTML, etc.

WWW: https://metacpan.org/release/WWW-Robot

Maintainer

The OpenBSD ports mailing-list

Categories

perl5 www

Build dependencies

Run dependencies

Test dependencies

Files

Ports Readmes 7.52, created on 2024-11-04, powered by Dancer 1.3521