Parsing :: Boost Libraries Documentation

# Parsing

A URL, short for "Uniform Resource Locator," is a compact string of characters identifying an abstract or physical resource. It has these five parts, which may be optional or disallowed depending on the context:

Each part’s syntax is defined by a set of production rules in rfc3986. All valid URLs conform to this grammar, also called the "generic syntax." Here is an example URL which describes a file and its location on a network host:

    https://www.example.com/path/to/file.txt?userid=1001&pages=3&results=full#page1

The parts and their corresponding text is as follows:

Part | Text
--- | ---
scheme | "https"
authority | "www.example.com"
path | "/path/to/file.txt"
query | "userid=1001&pages=3&results=full"
fragment | "page1"

The production rule for the example above is called a URI, which can contain all five parts. The specification using ABNF notation is:

    URI           = scheme ":" hier-part [ "?" query ] [ "#" fragment ]

    hier-part     = "//" authority path-abempty
                  / path-absolute
                  / path-rootless
                  / path-empty

In this notation, the square brackets ("\// [" and "\]") denote optional elements, quoted text represents character literals, and slashes are used to indicate a choice between one of several elements. For the complete specification of ABNF notation please consult rfc2234, "Augmented BNF for Syntax Specifications." When using this library to process or create URLs, it is necessary to choose which of these top-level production rules are applicable for a given use-case: absolute-URI, origin-form, relative-ref, URI, or URI-reference. These are discussed in greater depth later.

## Parsing URLs

Algorithms which parse URLs return a view which references the underlying character buffer without taking ownership, avoiding memory allocations and copies. The following example parses a string literal containing a URI:

    boost::core::string_view s = "https://user:pass@example.com:443/path/to/my%2dfile.txt?id=42&name=John%20Doe+Jingleheimer%2DSchmidt#page%20anchor";

The function returns a `result` which holds a `url_view` if the string is a valid URL. Otherwise it holds an `error_code`. It is impossible to construct a `url_view` which refers to an invalid URL.

| The caller is responsible for ensuring that the lifetime of the character buffer extends until it is no longer referenced by the view. These are the same semantics as that of `std::string_view`.

For convenience, a URL view can be constructed directly from the character buffer in a `string_view`. In this case, it parses the string according to the URI-reference grammar, throwing an exception upon failure. The following two statements are equivalent:

    boost::system::result<url_view> r = parse_uri( s );

In this library, free functions which parse things are named with the word "parse" followed by the name of the grammar used to match the string. There are several varieties of URLs, and depending on the use-case a particular grammar may be needed. In the target of an HTTP GET request for example, the scheme and fragment are omitted. This corresponds to the origin-form production rule described in rfc7230. The function `parse_origin_form` is suited for this purpose. All the URL parsing functions are listed here:

Function | Grammar | Example | Notes
--- | --- | --- | ---
`parse_absolute_uri` | absolute-URI | `http://www.boost.org/index.html?field=value` | No fragment
`parse_origin_form` | origin-form | `/index.html?field=value` | Used in HTTP
`parse_relative_ref` | relative-ref | `//www.boost.org/index.html?field=value#downloads` |
`parse_uri` | URI | `http://www.boost.org/index.html?field=value#downloads` |
`parse_uri_reference` | URI-reference | `http://www.boost.org/index.html` | Any URI or relative-ref

The URL is stored in its serialized form. Therefore, it can always be easily output, sent, or embedded as part of a protocol:

    url u = parse_uri_reference( "https://www.example.com/path/to/file.txt" ).value();

    assert(u.encoded_path() == "/path/to/file.txt");

A `url` is an allocating container which owns its character buffer. Upon construction from `url_view`, it allocates dynamic storage to hold a copy of the string.

    boost::system::result< url > rv = parse_uri_reference( "https://www.example.com/path/to/file.txt" );

    static_assert( std::is_convertible< boost::system::result< url_view >, boost::system::result< url > >::value, "" );

A `static_url` is a container which owns its character buffer for a URL whose maximum size is known. Upon construction from `url_view`, it does not perform any dynamic memory allocations.

    boost::system::result< static_url<1024> > rv = parse_uri_reference( "https://www.example.com/path/to/file.txt" );

    static_assert( std::is_convertible< boost::system::result< static_url<1024> >, boost::system::result< url > >::value, "" );

## Result Type

These functions have a return type which uses the `result` alias template. This class allows the parsing algorithms to report errors without referring to exceptions.

The functions `result::operator bool()` and `result::operator*` can be used to check if the result contains an error.

    boost::system::result< url > ru = parse_uri_reference( "https://www.example.com/path/to/file.txt" );
    if ( ru )
    {
        url u = *ru;
        assert(u.encoded_path() == "/path/to/file.txt");
    }
    else
    {
        boost::system::error_code e = ru.error();
        handle_error(e);
    }

Since `result::operator bool()` is already checking if `result` contains an error, `result::operator*` provides an unchecked alternative to get a value from `result`. In contexts where it is acceptable to throw errors, `result::value` can be used directly.

    try
    {
        url u = parse_uri_reference( "https://www.example.com/path/to/file.txt" ).value();
        assert(u.encoded_path() == "/path/to/file.txt");
    }
    catch (boost::system::system_error &e)
    {
        handle_error(e);
    }

Check the reference for `result` for a synopsis of the type.

## Constexpr Parsing

In C++20 and later, all parse functions are `constexpr`. This means URLs can be parsed and validated at compile time, which is useful when one side of a comparison is a fixed string that should be pre-parsed and checked for correctness.

For instance, a `constexpr` `url_view` parsed from a valid string literal is fully resolved at compile time. If the literal is malformed, `value()` attempts to throw, which is not permitted during constant evaluation, so the program fails to compile:

    // Parse and validate a URL at compile time (C++20).
    // value() on an error produces a compile error,
    // so invalid URLs are caught at build time.
    constexpr url_view api_base =
        parse_uri("https://api.example.com/v2").value();

    // A malformed literal would fail to compile:
    // constexpr url_view bad =
    //     parse_uri("ht tp://bad url").value();

Pre-parsed `constexpr` URL views can also serve as zero-cost constants. All parsing is done at compile time; at runtime the components are accessed with no parsing overhead:

    // Pre-parsed URL constants have zero runtime cost.
    // Parsing happens entirely at compile time.
    constexpr url_view default_endpoint =
        parse_uri("https://api.example.com:8443/v1").value();
    constexpr url_view fallback_endpoint =
        parse_uri("http://localhost:9090/debug").value();

    // Components are available at runtime with no parsing overhead
    assert(default_endpoint.port_number() == 8443);
    assert(fallback_endpoint.scheme() == "http");

Because `url_view` does not own its character buffer, `constexpr` URL views always reference string literals.

Edit this Page