This allow to use base_url in robots.txt, to reference a sitemap for
example.

Closes #443

Fix #430

Not only is this tidier, but it avoids making these iterators
Rayon-specific.

This offers a big performance bump for large sites:

  small-kb (100 pages)
    before: 530ms
     after: 500ms

  medium-kb (1000 pages)
    before: 2.9s
     after: 1.5s

  huge-kb (10,000 pages)
    before: 150s
     after:   9.1s

Fix #383

Closes #388

Closes #417

Closes #403

Fix #412

As mentioned in #381, crates.io 404's any request without an Accept:
text/html header.  It 200's any request with one, but at least
false-successes don't prevent checking any other links.

This also makes it easier to add a custom User-Agent if desired.

rustfmt and fix a clippy nit (unnecessary return) while I'm here.

As mentioned in #376

Push summary handling into Markdown parsing, identifying the presence
of one by giving its length in the rendered markup.

Hopefully a better fix for #376.

Also fix tests

Close #367, #372

This reverts commit c7156a84f0.