You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

README.md 6.6KB

8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233
  1. [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
  2. ![beacon for google analytics](https://ga-beacon.appspot.com/UA-88834340-1/tantivy-cli/README)
  3. `tantivy-cli` is the project hosting the command line interface for [tantivy](https://github.com/tantivy-search/tantivy), a search engine project.
  4. # Tutorial: Indexing Wikipedia with Tantivy CLI
  5. ## Introduction
  6. In this tutorial, we will create a brand new index with the articles of English wikipedia in it.
  7. ## Installing the tantivy CLI.
  8. There are a couple ways to install `tantivy-cli`.
  9. If you are a Rust programmer, you probably have `cargo` installed and you can just
  10. run `cargo install tantivy-cli`
  11. ## Creating the index: `new`
  12. Let's create a directory in which your index will be stored.
  13. ```bash
  14. # create the directory
  15. mkdir wikipedia-index
  16. ```
  17. We will now initialize the index and create its schema.
  18. The [schema](https://tantivy-search.github.io/tantivy/tantivy/schema/index.html) defines
  19. the list of your fields, and for each field:
  20. - its name
  21. - its type, currently `u64`, `i64` or `str`
  22. - how it should be indexed.
  23. You can find more information about the latter on
  24. [tantivy's schema documentation page](https://tantivy-search.github.io/tantivy/tantivy/schema/index.html)
  25. In our case, our documents will contain
  26. * a title
  27. * a body
  28. * a url
  29. We want the title and the body to be tokenized and indexed. We also want
  30. to add the term frequency and term positions to our index.
  31. (To be honest, phrase queries are not yet implemented in tantivy,
  32. so the positions won't be really useful in this tutorial.)
  33. Running `tantivy new` will start a wizard that will help you
  34. define the schema of the new index.
  35. Like all the other commands of `tantivy`, you will have to
  36. pass it your index directory via the `-i` or `--index`
  37. parameter as follows:
  38. ```bash
  39. tantivy new -i wikipedia-index
  40. ```
  41. Answer the questions as follows:
  42. ```none
  43. Creating new index
  44. Let's define its schema!
  45. New field name ? title
  46. Text or unsigned 32-bit integer (T/I) ? T
  47. Should the field be stored (Y/N) ? Y
  48. Should the field be indexed (Y/N) ? Y
  49. Should the field be tokenized (Y/N) ? Y
  50. Should the term frequencies (per doc) be in the index (Y/N) ? Y
  51. Should the term positions (per doc) be in the index (Y/N) ? Y
  52. Add another field (Y/N) ? Y
  53. New field name ? body
  54. Text or unsigned 32-bit integer (T/I) ? T
  55. Should the field be stored (Y/N) ? Y
  56. Should the field be indexed (Y/N) ? Y
  57. Should the field be tokenized (Y/N) ? Y
  58. Should the term frequencies (per doc) be in the index (Y/N) ? Y
  59. Should the term positions (per doc) be in the index (Y/N) ? Y
  60. Add another field (Y/N) ? Y
  61. New field name ? url
  62. Text or unsigned 32-bit integer (T/I) ? T
  63. Should the field be stored (Y/N) ? Y
  64. Should the field be indexed (Y/N) ? N
  65. Add another field (Y/N) ? N
  66. [
  67. {
  68. "name": "title",
  69. "type": "text",
  70. "options": {
  71. "indexing": "position",
  72. "stored": true
  73. }
  74. },
  75. {
  76. "name": "body",
  77. "type": "text",
  78. "options": {
  79. "indexing": "position",
  80. "stored": true
  81. }
  82. },
  83. {
  84. "name": "url",
  85. "type": "text",
  86. "options": {
  87. "indexing": "unindexed",
  88. "stored": true
  89. }
  90. }
  91. ]
  92. ```
  93. After the wizard has finished, a `meta.json` should exist in `wikipedia-index/meta.json`.
  94. It is a fairly human readable JSON, so you can check its content.
  95. It contains two sections:
  96. - segments (currently empty, but we will change that soon)
  97. - schema
  98. # Indexing the document: `index`
  99. Tantivy's `index` command offers a way to index a json file.
  100. The file must contain one JSON object per line.
  101. The structure of this JSON object must match that of our schema definition.
  102. ```json
  103. {"body": "some text", "title": "some title", "url": "http://somedomain.com"}
  104. ```
  105. For this tutorial, you can download a corpus with the 5 million+ English Wikipedia articles in the right format here: [wiki-articles.json (2.34 GB)](https://www.dropbox.com/s/wwnfnu441w1ec9p/wiki-articles.json.bz2?dl=0).
  106. Make sure to decompress the file
  107. ```bash
  108. bunzip2 wiki-articles.json.bz2
  109. ```
  110. If you are in a rush you can [download 100 articles in the right format here (11 MB)](http://fulmicoton.com/tantivy-files/wiki-articles-1000.json).
  111. The `index` command will index your document.
  112. By default it will use as 3 thread, each with a buffer size of 1GB split a
  113. accross these threads.
  114. ```
  115. cat wiki-articles.json | tantivy index -i ./wikipedia-index
  116. ```
  117. You can change the number of threads by passing it the `-t` parameter, and the total
  118. buffer size used by the threads heap by using the `-m`. Note that tantivy's memory usage
  119. is greater than just this buffer size parameter.
  120. On my computer (8 core Xeon(R) CPU X3450 @ 2.67GHz), on 8 threads, indexing wikipedia takes around 9 minutes.
  121. While tantivy is indexing, you can peek at the index directory to check what is happening.
  122. ```bash
  123. ls ./wikipedia-index
  124. ```
  125. The main file is `meta.json`.
  126. You should also see a lot of files with a UUID as filename, and different extensions.
  127. Our index is in fact divided in segments. Each segment acts as an individual smaller index.
  128. Its name is simply a uuid.
  129. If you decided to index the complete wikipedia, you may also see some of these files disappear.
  130. Having too many segments can hurt search performance, so tantivy actually automatically starts
  131. merging segments.
  132. # Serve the search index: `serve`
  133. Tantivy's cli also embeds a search server.
  134. You can run it with the following command.
  135. ```
  136. tantivy serve -i wikipedia-index
  137. ```
  138. By default, it will serve on port `3000`.
  139. You can search for the top 20 most relevant documents for the query `Barack Obama` by accessing
  140. the following [url](http://localhost:3000/api/?q=barack+obama&nhits=20) in your browser
  141. http://localhost:3000/api/?q=barack+obama&nhits=20
  142. By default this query is treated as `barack OR obama`.
  143. You can also search for documents that contains both term, by adding a `+` sign before the terms in your query.
  144. http://localhost:3000/api/?q=%2Bbarack%20%2Bobama&nhits=20
  145. Also, `-` makes it possible to remove documents the documents containing a specific term.
  146. http://localhost:3000/api/?q=-barack%20%2Bobama&nhits=20
  147. Finally tantivy handle phrase queries.
  148. http://localhost:3000/api/?q=%22barack%20obama%22&nhits=20
  149. # Search the index via the command line
  150. You may also use the `search` command to stream all documents matching a specific query.
  151. The documents are returned in an unspecified order.
  152. ```
  153. tantivy search -i wikipedia-index -q "barack obama"
  154. ```