jstrong
/
zola
1
0
Fork 0
Code Issues 0 Pull Requests 0 Releases 0 Wiki Activity
Browse Source

Optionally do not slugify paths (#875) 

* maybe_slugify() only does simple sanitation if config.slugify is false

* slugify is disabled by default, turn on for backwards-compatibility

* First docs changes for optional slugification

* Remove # from slugs but not &

* Add/fix tests for utf8 slugs

* Fix test sites for i18n slugs

* fix templates tests for i18n slugs

* Rename slugify setting to slugify_paths

* Default slugify_paths

* Update documentation for slugify_paths

* quasi_slugify removes ?, /, # and newlines

* Remove forbidden NTFS chars in quasi_slugify()

* Slugification forbidden chars can be configured

* Remove trailing dot/space in quasi_slugify

* Fix NTFS path sanitation

* Revert configurable slugification charset

* Remove \r for windows newlines and \t tabulations in quasi_slugify()

* Update docs for output paths

* Replace slugify with slugify_paths

* Fix test

* Default to not slugifying

* Move slugs utils to utils crate

* Use slugify_paths for anchors as well
index-subcmd
Vincent Prouillet 4 years ago
parent
0a0b6a3ad4
commit
ceb9bc8ed7
21 changed files with 515 additions and 39 deletions
Split View
Diff Options
Show Stats Download Patch File Download Diff File
  1. +2
    -0
      .gitignore
  2. +2
    -0
      CHANGELOG.md
  3. +7
    -9
      Cargo.lock
  4. +3
    -0
      components/config/src/config.rs
  5. +0
    -1
      components/library/Cargo.toml
  6. +39
    -9
      components/library/src/content/page.rs
  7. +0
    -1
      components/library/src/lib.rs
  8. +248
    -3
      components/library/src/taxonomies/mod.rs
  9. +0
    -1
      components/rendering/Cargo.toml
  10. +0
    -1
      components/rendering/src/lib.rs
  11. +2
    -2
      components/rendering/src/markdown.rs
  12. +11
    -0
      components/rendering/tests/markdown.rs
  13. +4
    -2
      components/templates/src/global_fns/mod.rs
  14. +1
    -0
      components/utils/Cargo.toml
  15. +2
    -0
      components/utils/src/lib.rs
  16. +107
    -0
      components/utils/src/slugs.rs
  17. +5
    -3
      docs/content/documentation/content/linking.md
  18. +43
    -0
      docs/content/documentation/content/page.md
  19. +34
    -7
      docs/content/documentation/content/taxonomies.md
  20. +4
    -0
      docs/content/documentation/getting-started/configuration.md
  21. +1
    -0
      test_site/config.toml

+ 2
- 0
.gitignore View File

@@ -25,3 +25,5 @@ stage

# nixos dependencies snippet
shell.nix
# vim temporary files
**/.*.sw*

+ 2
- 0
CHANGELOG.md View File

@@ -5,6 +5,8 @@
### Breaking
- Remove `toc` variable in section/page context and pass it to `page.toc` and `section.toc` instead so they are
accessible everywhere
- [Slugification](https://en.wikipedia.org/wiki/Slug_(web_publishing)#Slug) of page paths is now optional. By default, every path will be slugified as it is happening right now.
To keep non-ASCII characters, set `slugify_paths = true` in your config.

### Other
- Add zenburn syntax highlighting theme


+ 7
- 9
Cargo.lock View File

@@ -344,10 +344,9 @@ source = "registry+https://github.com/rust-lang/crates.io-index"

[[package]]
name = "bincode"
version = "1.2.0"
version = "1.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"autocfg 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)",
"byteorder 1.3.2 (registry+https://github.com/rust-lang/crates.io-index)",
"serde 1.0.103 (registry+https://github.com/rust-lang/crates.io-index)",
]
@@ -1141,7 +1140,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"byteorder 1.3.2 (registry+https://github.com/rust-lang/crates.io-index)",
"gif 0.10.3 (registry+https://github.com/rust-lang/crates.io-index)",
"jpeg-decoder 0.1.16 (registry+https://github.com/rust-lang/crates.io-index)",
"jpeg-decoder 0.1.18 (registry+https://github.com/rust-lang/crates.io-index)",
"num-iter 0.1.39 (registry+https://github.com/rust-lang/crates.io-index)",
"num-rational 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
"num-traits 0.2.10 (registry+https://github.com/rust-lang/crates.io-index)",
@@ -1223,7 +1222,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"

[[package]]
name = "jpeg-decoder"
version = "0.1.16"
version = "0.1.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"byteorder 1.3.2 (registry+https://github.com/rust-lang/crates.io-index)",
@@ -1275,7 +1274,6 @@ dependencies = [
"serde 1.0.103 (registry+https://github.com/rust-lang/crates.io-index)",
"serde_derive 1.0.103 (registry+https://github.com/rust-lang/crates.io-index)",
"slotmap 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)",
"slug 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)",
"tempfile 3.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
"tera 1.0.0 (registry+https://github.com/rust-lang/crates.io-index)",
"toml 0.5.5 (registry+https://github.com/rust-lang/crates.io-index)",
@@ -2099,7 +2097,6 @@ dependencies = [
"regex 1.3.1 (registry+https://github.com/rust-lang/crates.io-index)",
"serde 1.0.103 (registry+https://github.com/rust-lang/crates.io-index)",
"serde_derive 1.0.103 (registry+https://github.com/rust-lang/crates.io-index)",
"slug 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)",
"syntect 3.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
"templates 0.1.0",
"tera 1.0.0 (registry+https://github.com/rust-lang/crates.io-index)",
@@ -2520,7 +2517,7 @@ name = "syntect"
version = "3.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"bincode 1.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
"bincode 1.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
"bitflags 1.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
"flate2 1.0.13 (registry+https://github.com/rust-lang/crates.io-index)",
"fnv 1.0.6 (registry+https://github.com/rust-lang/crates.io-index)",
@@ -3009,6 +3006,7 @@ version = "0.1.0"
dependencies = [
"errors 0.1.0",
"serde 1.0.103 (registry+https://github.com/rust-lang/crates.io-index)",
"slug 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)",
"tempfile 3.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
"tera 1.0.0 (registry+https://github.com/rust-lang/crates.io-index)",
"toml 0.5.5 (registry+https://github.com/rust-lang/crates.io-index)",
@@ -3259,7 +3257,7 @@ dependencies = [
"checksum backtrace-sys 0.1.32 (registry+https://github.com/rust-lang/crates.io-index)" = "5d6575f128516de27e3ce99689419835fce9643a9b215a14d2b5b685be018491"
"checksum base64 0.10.1 (registry+https://github.com/rust-lang/crates.io-index)" = "0b25d992356d2eb0ed82172f5248873db5560c4721f564b13cb5193bda5e668e"
"checksum base64 0.11.0 (registry+https://github.com/rust-lang/crates.io-index)" = "b41b7ea54a0c9d92199de89e20e58d49f02f8e699814ef3fdf266f6f748d15c7"
"checksum bincode 1.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "b8ab639324e3ee8774d296864fbc0dbbb256cf1a41c490b94cba90c082915f92"
"checksum bincode 1.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "5753e2a71534719bf3f4e57006c3a4f0d2c672a4b676eec84161f763eca87dbf"
"checksum bitflags 1.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "cf1de2fe8c75bc145a2f577add951f8134889b4795d47466a54a5c846d691693"
"checksum block-buffer 0.7.3 (registry+https://github.com/rust-lang/crates.io-index)" = "c0940dc441f31689269e10ac70eb1002a3a1d3ad1390e030043662eb7fe4688b"
"checksum block-padding 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)" = "fa79dedbb091f449f1f39e53edf88d5dbe95f895dae6135a8d7b881fb5af73f5"
@@ -3351,7 +3349,7 @@ dependencies = [
"checksum iovec 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)" = "b2b3ea6ff95e175473f8ffe6a7eb7c00d054240321b84c57051175fe3c1e075e"
"checksum ipconfig 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "aa79fa216fbe60834a9c0737d7fcd30425b32d1c58854663e24d4c4b328ed83f"
"checksum itoa 0.4.4 (registry+https://github.com/rust-lang/crates.io-index)" = "501266b7edd0174f8530248f87f99c88fbe60ca4ef3dd486835b8d8d53136f7f"
"checksum jpeg-decoder 0.1.16 (registry+https://github.com/rust-lang/crates.io-index)" = "c1aae18ffeeae409c6622c3b6a7ee49792a7e5a062eea1b135fbb74e301792ba"
"checksum jpeg-decoder 0.1.18 (registry+https://github.com/rust-lang/crates.io-index)" = "0256f0aec7352539102a9efbcb75543227b7ab1117e0f95450023af730128451"
"checksum kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)" = "7507624b29483431c0ba2d82aece8ca6cdba9382bff4ddd0f7490560c056098d"
"checksum language-tags 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)" = "a91d884b6667cd606bb5a69aa0c99ba811a115fc68915e7056ec08a46e93199a"
"checksum lazy_static 1.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646"


+ 3
- 0
components/config/src/config.rs View File

@@ -130,6 +130,8 @@ pub struct Config {
/// key into different language.
translations: HashMap<String, TranslateTerm>,

/// Whether to slugify page and taxonomy URLs (disable for UTF-8 URLs)
pub slugify_paths: bool,
/// Whether to highlight all code blocks found in markdown files. Defaults to false
pub highlight_code: bool,
/// Which themes to use for code highlighting. See Readme for supported themes
@@ -354,6 +356,7 @@ impl Default for Config {
title: None,
description: None,
theme: None,
slugify_paths: true,
highlight_code: false,
highlight_theme: "base16-ocean-dark".to_string(),
default_language: "en".to_string(),


+ 0
- 1
components/library/Cargo.toml View File

@@ -10,7 +10,6 @@ chrono = { version = "0.4", features = ["serde"] }
tera = "1"
serde = "1"
serde_derive = "1"
slug = "0.1"
regex = "1"
lazy_static = "1"



+ 39
- 9
components/library/src/content/page.rs View File

@@ -4,7 +4,6 @@ use std::path::{Path, PathBuf};

use regex::Regex;
use slotmap::DefaultKey;
use slug::slugify;
use tera::{Context as TeraContext, Tera};

use config::Config;
@@ -19,6 +18,7 @@ use utils::templates::render_template;
use content::file_info::FileInfo;
use content::has_anchor;
use content::ser::SerializingPage;
use utils::slugs::maybe_slugify_paths;

lazy_static! {
// Based on https://regex101.com/r/H2n38Z/1/tests
@@ -160,21 +160,21 @@ impl Page {

page.slug = {
if let Some(ref slug) = page.meta.slug {
slugify(&slug.trim())
maybe_slugify_paths(&slug.trim(), config.slugify_paths)
} else if page.file.name == "index" {
if let Some(parent) = page.file.path.parent() {
if let Some(slug) = slug_from_dated_filename {
slugify(&slug)
maybe_slugify_paths(&slug, config.slugify_paths)
} else {
slugify(parent.file_name().unwrap().to_str().unwrap())
maybe_slugify_paths(parent.file_name().unwrap().to_str().unwrap(), config.slugify_paths)
}
} else {
slugify(&page.file.name)
maybe_slugify_paths(&page.file.name, config.slugify_paths)
}
} else if let Some(slug) = slug_from_dated_filename {
slugify(&slug)
maybe_slugify_paths(&slug, config.slugify_paths)
} else {
slugify(&page.file.name)
maybe_slugify_paths(&page.file.name, config.slugify_paths)
}
};

@@ -443,7 +443,8 @@ Hello world"#;
slug = "hello-&-world"
+++
Hello world"#;
let config = Config::default();
let mut config = Config::default();
config.slugify_paths = true;
let res = Page::parse(Path::new("start.md"), content, &config, &PathBuf::new());
assert!(res.is_ok());
let page = res.unwrap();
@@ -452,6 +453,23 @@ Hello world"#;
assert_eq!(page.permalink, config.make_permalink("hello-world"));
}

#[test]
fn can_make_url_from_utf8_slug_frontmatter() {
let content = r#"
+++
slug = "日本"
+++
Hello world"#;
let mut config = Config::default();
config.slugify_paths = false;
let res = Page::parse(Path::new("start.md"), content, &config, &PathBuf::new());
assert!(res.is_ok());
let page = res.unwrap();
assert_eq!(page.path, "日本/");
assert_eq!(page.components, vec!["日本"]);
assert_eq!(page.permalink, config.make_permalink("日本"));
}

#[test]
fn can_make_url_from_path() {
let content = r#"
@@ -508,7 +526,8 @@ Hello world"#;

#[test]
fn can_make_slug_from_non_slug_filename() {
let config = Config::default();
let mut config = Config::default();
config.slugify_paths = true;
let res =
Page::parse(Path::new(" file with space.md"), "+++\n+++", &config, &PathBuf::new());
assert!(res.is_ok());
@@ -517,6 +536,17 @@ Hello world"#;
assert_eq!(page.permalink, config.make_permalink(&page.slug));
}

#[test]
fn can_make_path_from_utf8_filename() {
let mut config = Config::default();
config.slugify_paths = false;
let res = Page::parse(Path::new("日本.md"), "+++\n++++", &config, &PathBuf::new());
assert!(res.is_ok());
let page = res.unwrap();
assert_eq!(page.slug, "日本");
assert_eq!(page.permalink, config.make_permalink(&page.slug));
}

#[test]
fn can_specify_summary() {
let config = Config::default();


+ 0
- 1
components/library/src/lib.rs View File

@@ -1,5 +1,4 @@
extern crate serde;
extern crate slug;
extern crate tera;
#[macro_use]
extern crate serde_derive;


+ 248
- 3
components/library/src/taxonomies/mod.rs View File

@@ -1,7 +1,6 @@
use std::collections::HashMap;

use slotmap::DefaultKey;
use slug::slugify;
use tera::{Context, Tera};

use config::{Config, Taxonomy as TaxonomyConfig};
@@ -10,6 +9,7 @@ use utils::templates::render_template;

use content::SerializingPage;
use library::Library;
use utils::slugs::maybe_slugify_paths;
use sorting::sort_pages_by_date;

#[derive(Debug, Clone, PartialEq, Serialize)]
@@ -69,7 +69,7 @@ impl TaxonomyItem {
})
.collect();
let (mut pages, ignored_pages) = sort_pages_by_date(data);
let slug = slugify(name);
let slug = maybe_slugify_paths(name, config.slugify_paths);
let permalink = if taxonomy.lang != config.default_language {
config.make_permalink(&format!("/{}/{}/{}", taxonomy.lang, taxonomy.name, slug))
} else {
@@ -169,7 +169,6 @@ impl Taxonomy {
self.items.iter().map(|i| SerializedTaxonomyItem::from_item(i, library)).collect();
context.insert("terms", &terms);
context.insert("taxonomy", &self.kind);
context.insert("lang", &self.kind.lang);
context.insert("current_url", &config.make_permalink(&self.kind.name));
context.insert("current_path", &self.kind.name);

@@ -331,6 +330,101 @@ mod tests {
assert_eq!(categories.items[1].pages.len(), 1);
}

#[test]
fn can_make_slugified_taxonomies() {
let mut config = Config::default();
let mut library = Library::new(2, 0, false);

config.taxonomies = vec![
TaxonomyConfig {
name: "categories".to_string(),
lang: config.default_language.clone(),
..TaxonomyConfig::default()
},
TaxonomyConfig {
name: "tags".to_string(),
lang: config.default_language.clone(),
..TaxonomyConfig::default()
},
TaxonomyConfig {
name: "authors".to_string(),
lang: config.default_language.clone(),
..TaxonomyConfig::default()
},
];

let mut page1 = Page::default();
let mut taxo_page1 = HashMap::new();
taxo_page1.insert("tags".to_string(), vec!["rust".to_string(), "db".to_string()]);
taxo_page1.insert("categories".to_string(), vec!["Programming tutorials".to_string()]);
page1.meta.taxonomies = taxo_page1;
page1.lang = config.default_language.clone();
library.insert_page(page1);

let mut page2 = Page::default();
let mut taxo_page2 = HashMap::new();
taxo_page2.insert("tags".to_string(), vec!["rust".to_string(), "js".to_string()]);
taxo_page2.insert("categories".to_string(), vec!["Other".to_string()]);
page2.meta.taxonomies = taxo_page2;
page2.lang = config.default_language.clone();
library.insert_page(page2);

let mut page3 = Page::default();
let mut taxo_page3 = HashMap::new();
taxo_page3.insert("tags".to_string(), vec!["js".to_string()]);
taxo_page3.insert("authors".to_string(), vec!["Vincent Prouillet".to_string()]);
page3.meta.taxonomies = taxo_page3;
page3.lang = config.default_language.clone();
library.insert_page(page3);

let taxonomies = find_taxonomies(&config, &library).unwrap();
let (tags, categories, authors) = {
let mut t = None;
let mut c = None;
let mut a = None;
for x in taxonomies {
match x.kind.name.as_ref() {
"tags" => t = Some(x),
"categories" => c = Some(x),
"authors" => a = Some(x),
_ => unreachable!(),
}
}
(t.unwrap(), c.unwrap(), a.unwrap())
};
assert_eq!(tags.items.len(), 3);
assert_eq!(categories.items.len(), 2);
assert_eq!(authors.items.len(), 1);

assert_eq!(tags.items[0].name, "db");
assert_eq!(tags.items[0].slug, "db");
assert_eq!(tags.items[0].permalink, "http://a-website.com/tags/db/");
assert_eq!(tags.items[0].pages.len(), 1);

assert_eq!(tags.items[1].name, "js");
assert_eq!(tags.items[1].slug, "js");
assert_eq!(tags.items[1].permalink, "http://a-website.com/tags/js/");
assert_eq!(tags.items[1].pages.len(), 2);

assert_eq!(tags.items[2].name, "rust");
assert_eq!(tags.items[2].slug, "rust");
assert_eq!(tags.items[2].permalink, "http://a-website.com/tags/rust/");
assert_eq!(tags.items[2].pages.len(), 2);

assert_eq!(categories.items[0].name, "Other");
assert_eq!(categories.items[0].slug, "other");
assert_eq!(categories.items[0].permalink, "http://a-website.com/categories/other/");
assert_eq!(categories.items[0].pages.len(), 1);

assert_eq!(categories.items[1].name, "Programming tutorials");
assert_eq!(categories.items[1].slug, "programming-tutorials");
assert_eq!(
categories.items[1].permalink,
"http://a-website.com/categories/programming-tutorials/"
);
assert_eq!(categories.items[1].pages.len(), 1);
}

#[test]
fn errors_on_unknown_taxonomy() {
let mut config = Config::default();
@@ -466,4 +560,155 @@ mod tests {
);
assert_eq!(categories.items[1].pages.len(), 1);
}

#[test]
fn can_make_utf8_taxonomies() {
let mut config = Config::default();
config.slugify_paths = false;
config.languages.push(Language {
rss: false,
code: "fr".to_string(),
..Language::default()
});
let mut library = Library::new(2, 0, true);

config.taxonomies = vec![TaxonomyConfig {
name: "catégories".to_string(),
lang: "fr".to_string(),
..TaxonomyConfig::default()
}];

let mut page = Page::default();
page.lang = "fr".to_string();
let mut taxo_page = HashMap::new();
taxo_page.insert("catégories".to_string(), vec!["Écologie".to_string()]);
page.meta.taxonomies = taxo_page;
library.insert_page(page);

let taxonomies = find_taxonomies(&config, &library).unwrap();
let categories = &taxonomies[0];

assert_eq!(categories.items.len(), 1);
assert_eq!(categories.items[0].name, "Écologie");
assert_eq!(
categories.items[0].permalink,
"http://a-website.com/fr/catégories/Écologie/"
);
assert_eq!(categories.items[0].pages.len(), 1);
}

#[test]
fn can_make_slugified_taxonomies_in_multiple_languages() {
let mut config = Config::default();
config.slugify_paths = true;
config.languages.push(Language {
rss: false,
code: "fr".to_string(),
..Language::default()
});
let mut library = Library::new(2, 0, true);

config.taxonomies = vec![
TaxonomyConfig {
name: "categories".to_string(),
lang: config.default_language.clone(),
..TaxonomyConfig::default()
},
TaxonomyConfig {
name: "tags".to_string(),
lang: config.default_language.clone(),
..TaxonomyConfig::default()
},
TaxonomyConfig {
name: "auteurs".to_string(),
lang: "fr".to_string(),
..TaxonomyConfig::default()
},
TaxonomyConfig {
name: "tags".to_string(),
lang: "fr".to_string(),
..TaxonomyConfig::default()
},
];

let mut page1 = Page::default();
let mut taxo_page1 = HashMap::new();
taxo_page1.insert("tags".to_string(), vec!["rust".to_string(), "db".to_string()]);
taxo_page1.insert("categories".to_string(), vec!["Programming tutorials".to_string()]);
page1.meta.taxonomies = taxo_page1;
page1.lang = config.default_language.clone();
library.insert_page(page1);

let mut page2 = Page::default();
let mut taxo_page2 = HashMap::new();
taxo_page2.insert("tags".to_string(), vec!["rust".to_string()]);
taxo_page2.insert("categories".to_string(), vec!["Other".to_string()]);
page2.meta.taxonomies = taxo_page2;
page2.lang = config.default_language.clone();
library.insert_page(page2);

let mut page3 = Page::default();
page3.lang = "fr".to_string();
let mut taxo_page3 = HashMap::new();
taxo_page3.insert("tags".to_string(), vec!["rust".to_string()]);
taxo_page3.insert("auteurs".to_string(), vec!["Vincent Prouillet".to_string()]);
page3.meta.taxonomies = taxo_page3;
library.insert_page(page3);

let taxonomies = find_taxonomies(&config, &library).unwrap();
let (tags, categories, authors) = {
let mut t = None;
let mut c = None;
let mut a = None;
for x in taxonomies {
match x.kind.name.as_ref() {
"tags" => {
if x.kind.lang == "en" {
t = Some(x)
}
}
"categories" => c = Some(x),
"auteurs" => a = Some(x),
_ => unreachable!(),
}
}
(t.unwrap(), c.unwrap(), a.unwrap())
};

assert_eq!(tags.items.len(), 2);
assert_eq!(categories.items.len(), 2);
assert_eq!(authors.items.len(), 1);

assert_eq!(tags.items[0].name, "db");
assert_eq!(tags.items[0].slug, "db");
assert_eq!(tags.items[0].permalink, "http://a-website.com/tags/db/");
assert_eq!(tags.items[0].pages.len(), 1);

assert_eq!(tags.items[1].name, "rust");
assert_eq!(tags.items[1].slug, "rust");
assert_eq!(tags.items[1].permalink, "http://a-website.com/tags/rust/");
assert_eq!(tags.items[1].pages.len(), 2);

assert_eq!(authors.items[0].name, "Vincent Prouillet");
assert_eq!(authors.items[0].slug, "vincent-prouillet");
assert_eq!(
authors.items[0].permalink,
"http://a-website.com/fr/auteurs/vincent-prouillet/"
);
assert_eq!(authors.items[0].pages.len(), 1);

assert_eq!(categories.items[0].name, "Other");
assert_eq!(categories.items[0].slug, "other");
assert_eq!(categories.items[0].permalink, "http://a-website.com/categories/other/");
assert_eq!(categories.items[0].pages.len(), 1);

assert_eq!(categories.items[1].name, "Programming tutorials");
assert_eq!(categories.items[1].slug, "programming-tutorials");
assert_eq!(
categories.items[1].permalink,
"http://a-website.com/categories/programming-tutorials/"
);
assert_eq!(categories.items[1].pages.len(), 1);
}

}

+ 0
- 1
components/rendering/Cargo.toml View File

@@ -7,7 +7,6 @@ authors = ["Vincent Prouillet <prouillet.vincent@gmail.com>"]
tera = { version = "1", features = ["preserve_order"] }
syntect = "=3.2.0"
pulldown-cmark = "0.6"
slug = "0.1"
serde = "1"
serde_derive = "1"
pest = "2"


+ 0
- 1
components/rendering/src/lib.rs View File

@@ -1,5 +1,4 @@
extern crate pulldown_cmark;
extern crate slug;
extern crate syntect;
extern crate tera;
#[macro_use]


+ 2
- 2
components/rendering/src/markdown.rs View File

@@ -1,6 +1,5 @@
use pulldown_cmark as cmark;
use regex::Regex;
use slug::slugify;
use syntect::easy::HighlightLines;
use syntect::html::{
start_highlighted_html_snippet, styled_line_to_highlighted_html, IncludeBackground,
@@ -13,6 +12,7 @@ use front_matter::InsertAnchor;
use table_of_contents::{make_table_of_contents, Heading};
use utils::site::resolve_internal_link;
use utils::vec::InsertMany;
use utils::slugs::maybe_slugify_anchors;

use self::cmark::{Event, LinkType, Options, Parser, Tag};

@@ -298,7 +298,7 @@ pub fn markdown_to_html(content: &str, context: &RenderContext) -> Result<Render
let title = get_text(&events[start_idx + 1..end_idx]);
let id = heading_ref
.id
.unwrap_or_else(|| find_anchor(&inserted_anchors, slugify(&title), 0));
.unwrap_or_else(|| find_anchor(&inserted_anchors, maybe_slugify_anchors(&title, context.config.slugify_paths), 0));
inserted_anchors.push(id.clone());

// insert `id` to the tag


+ 11
- 0
components/rendering/tests/markdown.rs View File

@@ -351,6 +351,17 @@ fn can_add_id_to_headings_same_slug() {
assert_eq!(res.body, "<h1 id=\"hello\">Hello</h1>\n<h1 id=\"hello-1\">Hello</h1>\n");
}

#[test]
fn can_add_non_slug_id_to_headings() {
let tera_ctx = Tera::default();
let permalinks_ctx = HashMap::new();
let mut config = Config::default();
config.slugify_paths = false;
let context = RenderContext::new(&tera_ctx, &config, "", &permalinks_ctx, InsertAnchor::None);
let res = render_content(r#"# L'écologie et vous"#, &context).unwrap();
assert_eq!(res.body, "<h1 id=\"L'écologie_et_vous\">L'écologie et vous</h1>\n");
}

#[test]
fn can_handle_manual_ids_on_headings() {
let tera_ctx = Tera::default();


+ 4
- 2
components/templates/src/global_fns/mod.rs View File

@@ -389,7 +389,8 @@ mod tests {

#[test]
fn can_get_taxonomy() {
let config = Config::default();
let mut config = Config::default();
config.slugify_paths = true;
let taxo_config = TaxonomyConfig {
name: "tags".to_string(),
lang: config.default_language.clone(),
@@ -466,7 +467,8 @@ mod tests {

#[test]
fn can_get_taxonomy_url() {
let config = Config::default();
let mut config = Config::default();
config.slugify_paths = true;
let taxo_config = TaxonomyConfig {
name: "tags".to_string(),
lang: config.default_language.clone(),


+ 1
- 0
components/utils/Cargo.toml View File

@@ -10,6 +10,7 @@ unicode-segmentation = "1.2"
walkdir = "2"
toml = "0.5"
serde = "1"
slug = "0.1"

[dev-dependencies]
tempfile = "3"

+ 2
- 0
components/utils/src/lib.rs View File

@@ -8,6 +8,7 @@ extern crate tera;
extern crate toml;
extern crate unicode_segmentation;
extern crate walkdir;
extern crate slug;

pub mod de;
pub mod fs;
@@ -15,3 +16,4 @@ pub mod net;
pub mod site;
pub mod templates;
pub mod vec;
pub mod slugs;

+ 107
- 0
components/utils/src/slugs.rs View File

@@ -0,0 +1,107 @@
fn strip_chars(s: &str, chars: &str) -> String {
let mut sanitized_string = s.to_string();
sanitized_string.retain( |c| !chars.contains(c));
sanitized_string
}

fn strip_invalid_paths_chars(s: &str) -> String {
// NTFS forbidden characters : https://gist.github.com/doctaphred/d01d05291546186941e1b7ddc02034d3
// Also we need to trim . from the end of filename
let trimmed = s.trim_end_matches(|c| c == ' ' || c == '.');
let cleaned = trimmed.replace(" ", "_");
// And () [] since they are not allowed in markdown links
strip_chars(&cleaned, "<>:/|?*#()[]\n\"\\\r\t")
}

fn strip_invalid_anchors_chars(s: &str) -> String {
// spaces are not valid in markdown links
let cleaned = s.replace(" ", "_");
// https://tools.ietf.org/html/rfc3986#section-3.5
strip_chars(&cleaned, "\"#%<>[\\]()^`{|}")
}

pub fn maybe_slugify_paths(s: &str, slugify: bool) -> String {
if slugify {
// ASCII slugification
slug::slugify(s)
}
else {
// Only remove forbidden characters
strip_invalid_paths_chars(s)
}
}

pub fn maybe_slugify_anchors(s: &str, slugify: bool) -> String {
if slugify {
// ASCII slugification
slug::slugify(s)
}
else {
// Only remove forbidden characters
strip_invalid_anchors_chars(s)
}
}

#[cfg(test)]
mod tests {
use super::*;

#[test]
fn strip_invalid_paths_chars_works() {
let tests = vec![
// no newlines
("test\ntest", "testtest"),
// no whitespaces
("test ", "test"),
("t est ", "t_est"),
// invalid NTFS
("test .", "test"),
("test. ", "test"),
("test#test/test?test", "testtesttesttest"),
// Invalid CommonMark chars in links
("test (hey)", "test_hey"),
("test (hey", "test_hey"),
("test hey)", "test_hey"),
("test [hey]", "test_hey"),
("test [hey", "test_hey"),
("test hey]", "test_hey"),
// UTF-8
("日本", "日本"),
];

for (input, expected) in tests {
assert_eq!(strip_invalid_paths_chars(&input), expected);
}
}

#[test]
fn strip_invalid_anchors_chars_works() {
let tests = vec![
("日本", "日本"),
// Some invalid chars get removed
("test#", "test"),
("test<", "test"),
("test%", "test"),
("test^", "test"),
("test{", "test"),
("test|", "test"),
("test(", "test"),
// Spaces are replaced by `_`
("test hey", "test_hey"),
];

for (input, expected) in tests {
assert_eq!(strip_invalid_anchors_chars(&input), expected);
}
}

#[test]
fn maybe_slugify_paths_enabled() {
assert_eq!(maybe_slugify_paths("héhé", true), "hehe");
}

#[test]
fn maybe_slugify_paths_disabled() {
assert_eq!(maybe_slugify_paths("héhé", false), "héhé");
}
}

+ 5
- 3
docs/content/documentation/content/linking.md View File

@@ -4,9 +4,11 @@ weight = 50
+++

## Heading id and anchor insertion
While rendering the Markdown content, a unique id will automatically be assigned to each heading. This id is created
by converting the heading text to a [slug](https://en.wikipedia.org/wiki/Semantic_URL#Slug), and appending numbers at
the end if the slug already exists for that article. For example:
While rendering the Markdown content, a unique id will automatically be assigned to each heading.
This id is created by converting the heading text to a [slug](https://en.wikipedia.org/wiki/Semantic_URL#Slug) if `slugify_paths` is enabled.
if `slugify_paths` is disabled, whitespaces are replaced by `_` and the following characters are stripped: `#`, `%`, `<`, `>`, `[`, `]`, `(`, `)`, \`, `^`, `{`, `|`, `}`.
A number is appended at the end if the slug already exists for that article
For example:

```md
# Something exciting! <- something-exciting


+ 43
- 0
docs/content/documentation/content/page.md View File

@@ -27,6 +27,49 @@ As you can see, creating an `about.md` file is equivalent to creating an
the `about` directory allows you to use asset co-location, as discussed in the
[overview](@/documentation/content/overview.md#asset-colocation) section.

## Output paths

For any page within your content folder, its output path will be defined by either:

- its `slug` frontmatter key
- its filename

Either way, these proposed path will be sanitized before being used.
If `slugify_paths` is enabled in the site's config - the default - paths are [slugified](https://en.wikipedia.org/wiki/Clean_URL#Slug).
Otherwise, a simpler sanitation is performed, outputting only valid NTFS paths.
The following characters are removed: `<`, `>`, `:`, `/`, `|`, `?`, `*`, `#`, `\\`, `(`, `)`, `[`, `]` as well as newlines and tabulations.
Additionally, trailing whitespace and dots are removed and whitespaces are replaced by `_`.

**NOTE:** To produce URLs containing non-English characters (UTF8), `slugify_paths` needs to be set to `false`.

### Path from frontmatter

The output path for the page will first be read from the `slug` key in the page's frontmatter.

**Example:** (file `content/zines/mlf-kurdistan.md`)

```
+++
title = "Le mouvement des Femmes Libres, à la tête de la libération kurde"
slug = "femmes-libres-libération-kurde"
+++
This is my article.
```

This frontmatter will output the article to `[base_url]/zines/femmes-libres-libération-kurde` with `slugify_paths` disabled, and to `[base_url]/zines/femmes-libres-liberation-kurde` with `slugify_enabled` enabled.

### Path from filename

When the article's output path is not specified in the frontmatter, it is extracted from the file's path in the content folder. Consider a file `content/foo/bar/thing.md`. The output path is constructed:
- if the filename is `index.md`, its parent folder name (`bar`) is used as output path
- otherwise, the output path is extracted from `thing` (the filename without the `.md` extension)

If the path found starts with a datetime string (`YYYY-mm-dd` or [a RFC3339 datetime](https://www.ietf.org/rfc/rfc3339.txt)) followed by an underscore (`_`) or a dash (`-`), this date is removed from the output path and will be used as the page date (unless already set in the front-matter). Note that the full RFC3339 datetime contains colons, which is not a valid character in a filename on Windows.

The output path extracted from the file path is then slugified or not depending on the `slugify_paths` config, as explained previously.

**Example:** The file `content/blog/2018-10-10-hello-world.md` will generated a page available at will be available at `[base_url]/hello-world`.

## Front matter

The TOML front matter is a set of metadata embedded in a file at the beginning of the file enclosed


+ 34
- 7
docs/content/documentation/content/taxonomies.md View File

@@ -5,7 +5,7 @@ weight = 90

Zola has built-in support for taxonomies.

The first step is to define the taxonomies in your [config.toml](@/documentation/getting-started/configuration.md).
## Configuration

A taxonomy has five variables:

@@ -16,21 +16,48 @@ For example the default would be page/1.
- `rss`: if set to `true`, an RSS feed will be generated for each term.
- `lang`: only set this if you are making a multilingual site and want to indicate which language this taxonomy is for

Once this is done, you can then set taxonomies in your content and Zola will pick
them up:
**Example 1:** (one language)

```toml
taxonomies = [ name = "categories", rss = true ]
```

**Example 2:** (multilingual site)

```toml
taxonomies = [
{name = "tags", lang = "fr"},
{name = "tags", lang = "eo"},
{name = "tags", lang = "en"},
]
```

## Using taxonomies

Once the configuration is done, you can then set taxonomies in your content and Zola will pick them up:

**Example:**

```toml
+++
...
title = "Writing a static-site generator in Rust"
date = 2019-08-15
[taxonomies]
tags = ["rust", "web"]
categories = ["programming"]
+++
```

The taxonomy pages are available at the following paths:
## Output paths

In a similar manner to how section and pages calculate their output path:
- the taxonomy name is never slugified
- the taxonomy entry (eg. as specific tag) is slugified when `slugify_paths` is enabled in the configuration

The taxonomy pages are then available at the following paths:

```plain
$BASE_URL/$NAME/
$BASE_URL/$NAME/$SLUG
$BASE_URL/$NAME/ (taxonomy)
$BASE_URL/$NAME/$SLUG (taxonomy entry)
```


+ 4
- 0
docs/content/documentation/getting-started/configuration.md View File

@@ -27,6 +27,10 @@ default_language = "en"
# The site theme to use.
theme = ""

# Slugify paths for compatibility with ASCII-only URLs produced by Zola < 0.9
# Enabling this setting removes non-English (UTF8) characters in URLs
slugify_paths = false

# When set to "true", all code blocks are highlighted.
highlight_code = false



+ 1
- 0
test_site/config.toml View File

@@ -4,6 +4,7 @@ highlight_code = true
compile_sass = true
generate_rss = true
theme = "sample"
slugify_paths = true

taxonomies = [
{name = "categories", rss = true},


Write Preview
Loading…
Cancel
Save