1 Commits

Author SHA1 Message Date
Alex Schroeder
761a5f16dd Use full text search 2023-09-10 23:41:11 +02:00
29 changed files with 863 additions and 860 deletions

201
LICENSE-2.0 Normal file
View File

@@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

View File

@@ -25,4 +25,4 @@ test:
upload:
go build
rsync --itemize-changes --archive oddmu oddmu.service *.html README.md sibirocobombus.root:/home/oddmu/
ssh sibirocobombus.root "systemctl restart oddmu; systemctl restart alex"
ssh sibirocobombus.root "systemctl restart oddmu"

203
README.md
View File

@@ -29,15 +29,11 @@ Oddmu. 🙃
## Markdown
This wiki uses a [Markdown
library](https://github.com/gomarkdown/markdown) to generate the web
pages from Markdown. There is no additional wiki markup. Most
importantly, double square brackets are not a link. If you're used to
This wiki uses Markdown. There is no additional wiki markup, most
importantly double square brackets are not a link. If you're used to
that, it'll be strange as you need to repeat the name: `[like
this](like this)`.
The Markdown processor comes with a few extensions, some of which are
enable by default:
this](like this)`. The Markdown processor comes with a few extensions,
some of which are enable by default:
* emphasis markers inside words are ignored
* tables are supported
@@ -50,6 +46,10 @@ enable by default:
* definition lists are supported
* MathJax is supported (but needs a separte setup)
See the section on
[extensions](https://github.com/gomarkdown/markdown#extensions) in the
Markdown library for information on the various extensions.
A table with footers and a columnspan:
```text
@@ -71,66 +71,28 @@ Internet
: Vector of transmission for pictures of cats
```
There is another extension made: hashtags link to searches for the
hashtag. Hashtags are separate from titles because there is no space
after the hash. Use the underscore to use hashtags consisting of
multiple words.
```
# Title
Text
#Tag #Another_Tag
```
## Templates
The template files are the HTML files in the working directory:
`add.html`, `edit.html`, `search.html`, `upload.html` and `view.html`.
Feel free to change the templates and restart the server. The first
change you should make is to replace the email address in `view.html`.
😄
Feel free to change the templates `view.html` and `edit.html` and
restart the server. Modifying the styles in the templates would be a
good start to get a feel for it.
See [Structuring the web
with HTML](https://developer.mozilla.org/en-US/docs/Learn/HTML) to
learn more about HTML.
Modifying the styles in the templates would be another good start to
get a feel for it. See [Learn to style HTML using
CSS](https://developer.mozilla.org/en-US/docs/Learn/CSS) to learn more
about style sheets.
The first change you should make is to replace the email address in
`view.html`. 😄
The templates can refer to the following properties of a page:
`{{.Title}}` is the page title. If the page doesn't provide its own
title, the page name is used.
`{{.Name}}` is the page name, escaped for use in URLs. More
specifically, it is URI escaped except for the slashes. The page name
doesn't include the `.md` extension.
`{{.Name}}` is the page name. The page name doesn't include the `.md`
extension.
`{{.Html}}` is the rendered Markdown, as HTML.
`{{printf "%s" .Body}}` is the Markdown, as a string (the data itself
is a byte array and that's why we need to call `printf`).
For the `search.html` template only:
`{{.Results}}` indicates if there were any search results.
`{{.Items}}` is an array of pages, each containing a search result. A
search result is a page (with the properties seen above). Thus, to
refer to them, you need to use a `{{range .Items}}``{{end}}`
construct.
For search results, `{{.Html}}` is the rendered Markdown of a page
summary, as HTML.
`{{.Score}}` is a numerical score for search results.
The `upload.html` template cannot refer to anything.
When calling the `save` action, the page name is take from the URL and
the page content is taken from the `body` form parameter. To
illustrate, here's how to edit a page using `curl`:
@@ -140,11 +102,6 @@ curl --form body="Did you bring a towel?" \
http://localhost:8080/save/welcome
```
The wiki uses the standard
[html/template](https://pkg.go.dev/html/template) library to do this.
There's more information on writing templates in the documentation for
the [text/template](https://pkg.go.dev/text/template) library.
## Non-English hyphenation
Automatic hyphenation by the browser requires two things: The style
@@ -153,11 +110,12 @@ and that element must have a `lang` set (usually a two letter language
code such as `de` for German). This happens in the template files,
such as `view.html` and `search.html`.
Oddmu uses the [lingua](github.com/pemistahl/lingua-go) library to
detect languages. If you know that you're only going to use a small
number of languages or just a single language! you can set the
environment variable ODDMU_LANGUAGES to a comma-separated list of ISO
639-1 codes, e.g. "en" or "en,de,fr,pt".
If have pages in different languages, the problem is that they all use
the same template and that's not good. In such cases, it might be
better to not specificy the `lang` attribute in the template. This
also disables hyphenation by the browser, unfortunately. It might
still be better than using English hyphenation patterns for
non-English languages.
## Building
@@ -184,9 +142,6 @@ If you ran it in the source directory, try
http://localhost:8080/view/README this serves the README file you're
currently reading.
You can change the port by setting the ODDMU_PORT environment
variable.
## Deploying it using systemd
As root, on your server:
@@ -196,7 +151,7 @@ adduser --system --home /home/oddmu oddmu
```
Copy all the files into `/home/oddmu` to your server: `oddmu`,
`oddmu.service`, and all the template files ending in `.html`.
`oddmu.service`, `view.html` and `edit.html`.
Edit the `oddmu.service` file. These are the three lines you most
likely have to take care of:
@@ -205,7 +160,6 @@ likely have to take care of:
ExecStart=/home/oddmu/oddmu
WorkingDirectory=/home/oddmu
Environment="ODDMU_PORT=8080"
Environment="ODDMU_LANGUAGES=en,de"
```
Install the service file and enable it:
@@ -253,21 +207,27 @@ MDCertificateAgreement accepted
ServerAdmin alex@alexschroeder.ch
ServerName transjovian.org
SSLEngine on
ProxyPassMatch ^/(search|upload|save|(view|edit|save|add|append)/(.*))?$ http://localhost:8080/$1
RewriteEngine on
RewriteRule ^/$ http://%{HTTP_HOST}:8080/view/index [redirect]
RewriteRule ^/(view|edit|save|search)/(.*) http://%{HTTP_HOST}:8080/$1/$2 [proxy]
</VirtualHost>
```
First, it manages the domain, getting the necessary certificates. It
redirects regular HTTP traffic from port 80 to port 443. It turns on
the SSL engine for port 443. It proxies the requests for the wiki to
port 8080.
the SSL engine for port 443. It redirects `/` to `/view/index` and any
path that starts with `/view/`, `/edit/`, `/save/` or `/search/` is
proxied to port 8080 where the Oddmu program can handle it.
Thus, this is what happens:
* The user tells the browser to visit `transjovian.org`
* The browser sends a request for `http://transjovian.org` (on port 80)
* Apache redirects this to `https://transjovian.org/` by default (now on port 443)
* This is proxied to `http://transjovian.org:8080/` (no encryption, on port 8080)
* The user tells the browser to visit `http://transjovian.org` (on port 80)
* Apache redirects this to `http://transjovian.org/` by default (still on port 80)
* Our first virtual host redirects this to `https://transjovian.org/` (encrypted, on port 443)
* Our second virtual host redirects this to `https://transjovian.org/wiki/view/index` (still on port 443)
* This is proxied to `http://transjovian.org:8080/view/index` (no on port 8080, without encryption)
* The wiki converts `index.md` to HTML, adds it to the template, and serves it.
Restart the server, gracefully:
@@ -275,11 +235,6 @@ Restart the server, gracefully:
apachectl graceful
```
To serve both HTTP and HTTPS, don't redirect from the first virtual
host to the second instead just proxy to the wiki like you did for
the second virtual host: use a copy of the `ProxyPassMatch` directive
instead of `RewriteEngine on` and `RewriteRule`.
## Access
Access control is not part of the wiki. By default, the wiki is
@@ -310,12 +265,12 @@ To delete remove a user:
htpasswd -D .htpasswd berta
```
Modify your site configuration and protect the `/edit/`, `/save/`,
`/add/`, `/append/`, `/upload` and `/save` URLs with a password by
adding the following to your `<VirtualHost *:443>` section:
Modify your site configuration and protect the `/edit/` and `/save/`
URLs with a password by adding the following to your `<VirtualHost
*:443>` section:
```apache
<LocationMatch "^/(upload|save|(edit|save|add|append)/(.*))$">
<LocationMatch "^/(edit|save)/">
AuthType Basic
AuthName "Password Required"
AuthUserFile /home/oddmu/.htpasswd
@@ -340,9 +295,10 @@ webserver can read (world readable file, world readable and executable
directory). Populate it with files.
Make sure that none of the static files look like the wiki paths
`/view/`, `/edit/`, `/save/`, `/add/`, `/append/`, `/upload`, `/save`
or `/search`. For example, create a file called `robots.txt`
containing the following, tellin all robots that they're not welcome.
`/view/`, `/edit/`, `/save/` or `/search/`.
For example, create a file called `robots.txt` containing the
following, tellin all robots that they're not welcome.
```text
User-agent: *
@@ -366,7 +322,7 @@ above.
This requires a valid login by the user "alex" or "berta":
```apache
<LocationMatch "^/(edit|save|add|append)/intetebi/">
<LocationMatch "^/(edit|save)/intetebi/">
Require user alex berta
</LocationMatch>
```
@@ -387,14 +343,53 @@ that matches everything:
</Location>
```
## Virtual hosting
## Customization (with recompilation)
Virtual hosting in this context means that the program serves two
different sites for two different domains from the same machine. Oddmu
doesn't support that, but your webserver does. Therefore, start an
Oddmu instance for every domain name, each listening on a different
port. Then set up your web server such that ever domain acts as a
reverse proxy to a different Oddmu instance.
The Markdown parser can be customized and
[extensions](https://pkg.go.dev/github.com/gomarkdown/markdown/parser#Extensions)
can be added. There's an example in the
[usage](https://github.com/gomarkdown/markdown#usage) section. You'll
need to make changes to the `viewHandler` yourself.
### Render Gemtext
In a first approximation, Gemtext is valid Markdown except for the
rocket links (`=>`). Here's how to modify the `loadPage` so that a
`.gmi` file is loaded if no `.md` is found, and the rocket links are
translated into Markdown:
```go
func loadPage(name string) (*Page, error) {
filename := name + ".md"
body, err := os.ReadFile(filename)
if err == nil {
return &Page{Title: name, Name: name, Body: body}, nil
}
filename = name + ".gmi"
body, err = os.ReadFile(filename)
if err == nil {
re := regexp.MustCompile(`(?m)^=>\s*(\S+)\s+(.+)`)
body = []byte(re.ReplaceAllString(string(body), `* [$2]($1)`))
return &Page{Title: name, Name: name, Body: body}, nil
}
return nil, err
}
```
There is a small problem, however: By default, Markdown expects an
empty line before a list begins. The following change to `renderHtml`
uses the `NoEmptyLineBeforeBlock` extension for the parser:
```go
func (p* Page) renderHtml() {
// Here is where a new extension is added!
extensions := parser.CommonExtensions | parser.NoEmptyLineBeforeBlock
markdownParser := parser.NewWithExtensions(extensions)
maybeUnsafeHTML := markdown.ToHTML(p.Body, markdownParser, nil)
html := bluemonday.UGCPolicy().SanitizeBytes(maybeUnsafeHTML)
p.Html = template.HTML(html);
}
```
## Understanding search
@@ -426,28 +421,20 @@ A document with content "This is a test" when searched with the phrase
"this test" therefore gets a score of 8: the entire phrase does not
match but each word gets four points.
Trigrams are sometimes strange: In a text containing the words "main"
and "rail", a search for "mail" returns a match because the trigrams
"mai" and "ail" are found. In this situation, the result has a score
of 0.
Trigrams are sometimes strange: In a text containing the words
"software" and "#socialmedia", a search for "#software" returns a
result because the trigram "#so" is part of "#socialmedia".
## Limitations
Page titles are filenames with `.md` appended. If your filesystem
cannot handle it, it can't be a page name.
cannot handle it, it can't be a page title. Specifically, *no slashes*
in filenames.
The pages are indexed as the server starts and the index is kept in
memory. If you have a ton of pages, this surely wastes a lot of
memory.
Files may not end with a tilde (`~`) these are backup files.
You cannot edit uploaded files. If you upload a file called
`hello.txt` and attempt to edit it by using `/edit/hello.txt` you will
create a page with the name `hello.txt.md` instead.
You cannot delete uploaded files via the web.
## Bugs
If you spot any, [contact](https://alexschroeder.ch/wiki/Contact) me.

21
TODO.md
View File

@@ -1,6 +1,23 @@
Upload files should use path info so that we can use Apache to
restrict access to directories.
Easily prepend or append text for use with a mobile browser. Like
comments.
Upload files.
Automatically scale or process files.
Post by Delta Chat? That is, allow certain encrypted emails to post.
Convert the existing wiki.
Investigate how to run a multi-lingual wiki where an appropriate
template is used based on the language of the page. This is important
because the template needs to use the appropriate `lang` attribute for
hyphenation to work.
Investigate how to run a multi-linugual wiki where an appropriate
version of a page is served based on language preferences of the user.
This is a low priority issue since it's probably only of interest for
corporate or governmental sites.
Switch from trigram search to a simple full text search engine?
https://artem.krylysov.com/blog/2020/07/28/lets-build-a-full-text-search-engine/

View File

@@ -1,21 +0,0 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="format-detection" content="telephone=no">
<meta name="viewport" content="width=device-width">
<title>Add to {{.Title}}</title>
<style>
html { max-width: 70ch; padding: 2ch; margin: auto; color: #111; background: #ffe; }
form, textarea { width: 100%; }
</style>
</head>
<body>
<h1>Adding to {{.Title}}</h1>
<form action="/append/{{.Name}}" method="POST">
<textarea name="body" rows="20" cols="80" placeholder="Text" autofocus required></textarea>
<p><input type="submit" value="Add">
<a href="/view/{{.Name}}"><button>Cancel</button></a></p>
</form>
</body>
</html>

View File

@@ -1,25 +0,0 @@
package main
import (
"fmt"
"os"
)
func commands() {
if len(os.Args) == 3 && os.Args[1] == "html" {
p, err := loadPage(os.Args[2]);
if err != nil {
fmt.Println(err);
} else {
p.renderHtml();
fmt.Println(p.Html);
}
} else {
fmt.Printf("Unknown command: %v\n", os.Args[1:])
fmt.Print("Without any arguments, serves a wiki.\n")
fmt.Print(" Environment variable ODDMUSE_PORT controls the port.\n")
fmt.Print(" Environment variable ODDMUSE_LANGAUGES controls the languages detected.\n")
fmt.Print("html PAGENAME\n")
fmt.Print(" Print the HTML of the page.\n")
}
}

View File

@@ -1,14 +0,0 @@
package main
import (
"github.com/stretchr/testify/assert"
"testing"
)
// Use go test -race to see whether this is a race condition.
func TestLoadAndSearch(t *testing.T) {
go loadIndex()
q := "Oddµ"
pages := search(q)
assert.Zero(t, len(pages))
}

View File

@@ -12,12 +12,11 @@ form, textarea { width: 100%; }
</head>
<body>
<h1>Editing {{.Title}}</h1>
<form action="/save/{{.Name}}" method="POST">
<textarea name="body" rows="20" cols="80" placeholder="# Title
Text" autofocus>{{printf "%s" .Body}}</textarea>
<form action="/save/{{.Name}}" method="POST">
<div><textarea name="body" rows="20" cols="80">{{printf "%s" .Body}}</textarea></div>
<p><input type="submit" value="Save">
<a href="/view/{{.Name}}"><button type="button">Cancel</button></a></p>
<a href="/view/{{.Name}}"><button>Cancel</button></a></p>
</form>
</body>
</html>

32
filter.go Normal file
View File

@@ -0,0 +1,32 @@
// Copyright 2020 Artem Krylysov
// Copyright 2023 Alex Schroeder
//
// Licensed under the Apache License, Version 2.0 (the "License"); you
// may not use this file except in compliance with the License. You
// may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
//
// This code was originally copied from
// https://github.com/akrylysov/simplefts
// https://artem.krylysov.com/blog/2020/07/28/lets-build-a-full-text-search-engine/
package main
import (
"strings"
)
// lowercaseFilter returns a slice of tokens normalized to lower case.
func lowercaseFilter(tokens []string) []string {
r := make([]string, len(tokens))
for i, token := range tokens {
r[i] = strings.ToLower(token)
}
return r
}

33
filter_test.go Normal file
View File

@@ -0,0 +1,33 @@
// Copyright 2020 Artem Krylysov
// Copyright 2023 Alex Schroeder
//
// Licensed under the Apache License, Version 2.0 (the "License"); you
// may not use this file except in compliance with the License. You
// may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
//
// This code was originally copied from
// https://github.com/akrylysov/simplefts
// https://artem.krylysov.com/blog/2020/07/28/lets-build-a-full-text-search-engine/
package main
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestLowercaseFilter(t *testing.T) {
var (
in = []string{"Cat", "DOG", "fish"}
out = []string{"cat", "dog", "fish"}
)
assert.Equal(t, out, lowercaseFilter(in))
}

2
go.mod
View File

@@ -4,7 +4,7 @@ go 1.21.0
require (
github.com/dgryski/go-trigram v0.0.0-20160407183937-79ec494e1ad0
github.com/gomarkdown/markdown v0.0.0-20230912175223-14b07df9d538
github.com/gomarkdown/markdown v0.0.0-20230716120725-531d2d74bc12
github.com/microcosm-cc/bluemonday v1.0.25
github.com/pemistahl/lingua-go v1.4.0
github.com/stretchr/testify v1.8.4

5
go.sum
View File

@@ -7,11 +7,7 @@ github.com/dgryski/go-trigram v0.0.0-20160407183937-79ec494e1ad0/go.mod h1:qzKC/
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
github.com/gomarkdown/markdown v0.0.0-20230716120725-531d2d74bc12 h1:uK3X/2mt4tbSGoHvbLBHUny7CKiuwUip3MArtukol4E=
github.com/gomarkdown/markdown v0.0.0-20230716120725-531d2d74bc12/go.mod h1:JDGcbDT52eL4fju3sZ4TeHGsQwhG9nbDV21aMyhwPoA=
github.com/gomarkdown/markdown v0.0.0-20230912175223-14b07df9d538 h1:ePDpFu7l0QUV46/9A7icfL2wvIOzTJLCWh4RO2NECzE=
github.com/gomarkdown/markdown v0.0.0-20230912175223-14b07df9d538/go.mod h1:JDGcbDT52eL4fju3sZ4TeHGsQwhG9nbDV21aMyhwPoA=
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.8 h1:e6P7q2lk1O+qJJb4BtCQXlK8vWEO8V1ZeuEdJNOqZyg=
github.com/google/go-cmp v0.5.8/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/gorilla/css v1.0.0 h1:BQqNyPTi50JCFMTw/b67hByjMVXZRwGha6wxVGkeihY=
github.com/gorilla/css v1.0.0/go.mod h1:Dn721qIggHpt4+EFCcTLTU/vk5ySda2ReITrtgBl60c=
github.com/microcosm-cc/bluemonday v1.0.25 h1:4NEwSfiJ+Wva0VxN5B8OwMicaJvD8r9tlJWm9rtloEg=
@@ -32,7 +28,6 @@ golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8T
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
google.golang.org/protobuf v1.31.0 h1:g0LDEJHgrBl9N9r17Ru3sqWhkIx2NB67okBHPwC7hs8=
google.golang.org/protobuf v1.31.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

121
index.go Normal file
View File

@@ -0,0 +1,121 @@
// Copyright 2020 Artem Krylysov
// Copyright 2023 Alex Schroeder
//
// Licensed under the Apache License, Version 2.0 (the "License"); you
// may not use this file except in compliance with the License. You
// may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
//
// This code was originally copied from
// https://github.com/akrylysov/simplefts
// https://artem.krylysov.com/blog/2020/07/28/lets-build-a-full-text-search-engine/
package main
import(
"log"
"sort"
)
type docid uint
// next_id is the number of the next document added to the index
var next_id docid = 0
// index is an inverted index. It maps tokens to document ids. Use add
// to add a document to the index. Use remove to remove a document
// from the index. Remove the old document and add the new document to
// update the index.
type index map[string][]docid
// add adds a document to the index and returns the new document id.
// This limits the number of documents and updates that can happen
// during a particular run.
func (idx index) add(text string) docid {
id := next_id; next_id++
for _, token := range analyze(text) {
ids := idx[token]
if ids != nil && ids[len(ids)-1] == id {
// Don't add same ID twice.
continue
}
idx[token] = append(ids, id)
}
return id
}
// delete removes a document from the index by removing all the words.
func (idx index) delete(text string, id docid) {
for _, token := range analyze(text) {
ids := idx[token]
// If the token doesn't show up, that's strange. This
// shouldn't happen.
if ids == nil {
log.Printf("Token %s is not indexed", token)
continue
}
// If the token appears only in this document, remove
// the whole entry.
if len(ids) == 1 && ids[0] == id {
delete(idx, token)
continue
}
// Otherwise, remove the token from the index.
i := sort.Search(len(ids), func(i int) bool { return ids[i] >= id })
if i != -1 && i < len(ids) && ids[i] == id {
copy(ids[i:], ids[i+1:])
idx[token] = ids[:len(ids)-1]
continue
}
// If none of the above, then our docid wasn't
// indexed. This shouldn't happen, either.
log.Printf("The index for token %s does not contain doc id %d", token, id)
}
}
// intersection returns the set intersection between a and b.
// a and b have to be sorted in ascending order and contain no duplicates.
func intersection(a []docid, b []docid) []docid {
maxLen := len(a)
if len(b) > maxLen {
maxLen = len(b)
}
r := make([]docid, 0, maxLen)
var i, j int
for i < len(a) && j < len(b) {
if a[i] < b[j] {
i++
} else if a[i] > b[j] {
j++
} else {
r = append(r, a[i])
i++
j++
}
}
return r
}
// search queries the index for the given text.
func (idx index) search(text string) []docid {
var r []docid
for _, token := range analyze(text) {
if ids, ok := idx[token]; ok {
if r == nil {
r = ids
} else {
r = intersection(r, ids)
}
} else {
// Token doesn't exist.
return nil
}
}
return r
}

41
index_test.go Normal file
View File

@@ -0,0 +1,41 @@
// Copyright 2020 Artem Krylysov
// Copyright 2023 Alex Schroeder
//
// Licensed under the Apache License, Version 2.0 (the "License"); you
// may not use this file except in compliance with the License. You
// may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
//
// This code was originally copied from
// https://github.com/akrylysov/simplefts
// https://artem.krylysov.com/blog/2020/07/28/lets-build-a-full-text-search-engine/
package main
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestIndex(t *testing.T) {
idx := make(index)
assert.Nil(t, idx.search("foo"))
assert.Nil(t, idx.search("donut"))
id1 := idx.add("A donut on a glass plate. Only the donuts.")
assert.Equal(t, idx.search("donut"), []docid{id1})
assert.Equal(t, idx.search("DoNuts"), []docid{id1})
assert.Equal(t, idx.search("glass"), []docid{id1})
id2 := idx.add("donut is a donut")
assert.Equal(t, idx.search("donut"), []docid{id1, id2})
assert.Equal(t, idx.search("DoNuts"), []docid{id1})
assert.Equal(t, idx.search("glass"), []docid{id1})
}

View File

@@ -1,58 +0,0 @@
package main
import (
"errors"
"github.com/pemistahl/lingua-go"
"os"
"strings"
)
// getLangauges returns the environment variable ODDMU_LANGUAGES or
// all languages.
func getLanguages() ([]lingua.Language, error) {
v := os.Getenv("ODDMU_LANGUAGES")
if v == "" {
return lingua.AllLanguages(), nil
}
codes := strings.Split(v, ",")
if len(codes) == 1 {
return nil, errors.New("detection unnecessary")
}
var langs []lingua.Language
for _, lang := range codes {
langs = append(langs, lingua.GetLanguageFromIsoCode639_1(lingua.GetIsoCode639_1FromValue(lang)))
}
return langs, nil
}
// detector is the LanguageDetector initialized at startup by loadLanguages.
var detector lingua.LanguageDetector
// loadLanguages initializes the detector using the languages returned
// by getLanguages and returns the number of languages loaded.
func loadLanguages() int {
langs, err := getLanguages()
if err == nil {
detector = lingua.NewLanguageDetectorBuilder().
FromLanguages(langs...).
WithPreloadedLanguageModels().
WithLowAccuracyMode().
Build()
} else {
detector = nil
}
return len(langs)
}
// language returns the language used for a string, as a lower case
// ISO 639-1 string, e.g. "en" or "de".
func language(s string) string {
if detector == nil {
return os.Getenv("ODDMU_LANGUAGES")
}
if language, ok := detector.DetectLanguageOf(s); ok {
return strings.ToLower(language.IsoCode639_1().String())
}
return ""
}

View File

@@ -1,50 +0,0 @@
package main
import (
"github.com/stretchr/testify/assert"
"os"
"testing"
)
func TestAllLanguage(t *testing.T) {
os.Unsetenv("ODDMU_LANGUAGES")
loadLanguages()
l := language(`
My back hurts at night
My shoulders won't budge today
Winter bones I say`)
assert.Equal(t, "en", l)
}
func TestSomeLanguages(t *testing.T) {
os.Setenv("ODDMU_LANGUAGES", "en,de")
loadLanguages()
l := language(`
Kühle Morgenluft
Keine Amsel singt heute
Mensch im Dämmerlicht
`)
assert.Equal(t, "de", l)
}
func TestOneLanguages(t *testing.T) {
os.Setenv("ODDMU_LANGUAGES", "en")
loadLanguages()
l := language(`
Schwer wiegt die Luft hier
Atme ein, ermahn' ich mich
Erinnerungen
`)
assert.Equal(t, "en", l)
}
func TestWrongLanguages(t *testing.T) {
os.Setenv("ODDMU_LANGUAGES", "de,fr")
loadLanguages()
l := language(`
Something drifts down there
Head submerged oh god a man
Drowning as we stare
`)
assert.NotEqual(t, "en", l)
}

View File

@@ -17,6 +17,7 @@ Environment="ODDMU_PORT=8080"
ReadWritePaths=/home/oddmu
ProtectHostname=yes
RestrictSUIDSGID=yes
UMask=0077
RemoveIPC=yes
MemoryDenyWriteExecute=yes

76
page.go
View File

@@ -7,13 +7,28 @@ import (
"github.com/gomarkdown/markdown/ast"
"github.com/gomarkdown/markdown/parser"
"github.com/microcosm-cc/bluemonday"
"github.com/pemistahl/lingua-go"
"html/template"
"net/url"
"os"
"path/filepath"
"strings"
)
// languages is the list of languages the wiki understands. This is
// passed along to the template so that it can be added to the
// template which allows browsers to (maybe) do hyphenation correctly.
var languages = []lingua.Language{
lingua.English,
lingua.German,
}
// detector is built once based on the list languages.
var detector = lingua.NewLanguageDetectorBuilder().
FromLanguages(languages...).
WithPreloadedLanguageModels().
WithLowAccuracyMode().
Build()
// Page is a struct containing information about a single page. Title
// is the title extracted from the page content using titleRegexp.
// Name is the filename without extension (so a filename of "foo.md"
@@ -29,26 +44,14 @@ type Page struct {
Score int
}
// santize uses bluemonday to sanitize the HTML.
func sanitize(s string) template.HTML {
return template.HTML(bluemonday.UGCPolicy().Sanitize(s))
}
// santizeBytes uses bluemonday to sanitize the HTML.
func sanitizeBytes(bytes []byte) template.HTML {
return template.HTML(bluemonday.UGCPolicy().SanitizeBytes(bytes))
}
// nameEscape returns the page name safe for use in URLs. That is,
// percent escaping is used except for the slashes.
func nameEscape(s string) string {
parts := strings.Split(s, "/")
for i, part := range parts {
parts[i] = url.PathEscape(part)
}
return strings.Join(parts, "/")
}
// save saves a Page. The filename is based on the Page.Name and gets
// the ".md" extension. Page.Body is saved, without any carriage
// return characters ("\r"). The file permissions used are readable
@@ -57,20 +60,17 @@ func nameEscape(s string) string {
func (p *Page) save() error {
filename := p.Name + ".md"
s := bytes.ReplaceAll(p.Body, []byte{'\r'}, []byte{})
if len(s) == 0 {
return os.Remove(filename)
}
p.Body = s
p.updateIndex()
d := filepath.Dir(filename)
if d != "." {
err := os.MkdirAll(d, 0755)
err := os.MkdirAll(d, 0700)
if err != nil {
fmt.Printf("Creating directory %s failed", d)
return err
}
}
return os.WriteFile(filename, s, 0644)
return os.WriteFile(filename, s, 0600)
}
// loadPage loads a Page given a name. The filename loaded is that
@@ -101,33 +101,11 @@ func (p *Page) handleTitle(replace bool) {
}
}
func hashtag(p *parser.Parser, data []byte, offset int) (int, ast.Node) {
data = data[offset:]
i := 0
n := len(data)
for i < n && !parser.IsSpace(data[i]) {
i++
}
if i == 0 {
return 0, nil
}
link := &ast.Link{
Destination: append([]byte("/search?q=%23"), data[1:i]...),
Title: data[0:i],
}
text := bytes.ReplaceAll(data[0:i], []byte("_"), []byte(" "))
ast.AppendChild(link, &ast.Text{Leaf: ast.Leaf{Literal: text}})
return i, link
}
// renderHtml renders the Page.Body to HTML and sets Page.Html.
func (p *Page) renderHtml() {
parser := parser.New()
parser.RegisterInline('#', hashtag)
maybeUnsafeHTML := markdown.ToHTML(p.Body, parser, nil)
p.Name = nameEscape(p.Name)
maybeUnsafeHTML := markdown.ToHTML(p.Body, nil, nil)
p.Html = sanitizeBytes(maybeUnsafeHTML)
p.Language = language(p.plainText())
p.Language = p.language(p.plainText())
}
// plainText renders the Page.Body to plain text and returns it,
@@ -163,5 +141,17 @@ func (p *Page) summarize(q string) {
p.Score = score(q, string(p.Body)) + score(q, p.Title)
t := p.plainText()
p.Html = sanitize(snippets(q, t))
p.Language = language(t)
p.Language = p.language(t)
}
func (p *Page) language(s string) string {
if language, ok := detector.DetectLanguageOf(s); ok {
switch language {
case lingua.English:
return "en"
case lingua.German:
return "de"
}
}
return ""
}

View File

@@ -1,9 +1,8 @@
package main
import (
"github.com/stretchr/testify/assert"
"os"
"regexp"
"strings"
"testing"
)
@@ -13,10 +12,19 @@ My back aches for you
I sit, stare and type for hours
But yearn for blue sky`)}
p.handleTitle(false)
assert.Equal(t, "Ache", p.Title)
assert.Regexp(t, regexp.MustCompile("^# Ache"), string(p.Body))
if p.Title != "Ache" {
t.Logf("The page title was not extracted correctly: %s", p.Title)
t.Fail()
}
if !strings.HasPrefix(string(p.Body), "# Ache") {
t.Logf("The page title was removed: %s", p.Body)
t.Fail()
}
p.handleTitle(true)
assert.Regexp(t, regexp.MustCompile("^My back"), string(p.Body))
if !strings.HasPrefix(string(p.Body), "My back") {
t.Logf("The page title was not removed: %s", p.Body)
t.Fail()
}
}
func TestPagePlainText(t *testing.T) {
@@ -24,8 +32,12 @@ func TestPagePlainText(t *testing.T) {
The air will not come
To inhale is an effort
The summer heat kills`)}
s := p.plainText()
r := "Water The air will not come To inhale is an effort The summer heat kills"
assert.Equal(t, r, p.plainText())
if s != r {
t.Logf("The plain text version is wrong: %s", s)
t.Fail()
}
}
func TestPageHtml(t *testing.T) {
@@ -34,32 +46,17 @@ Silver leaves shine bright
They droop, boneless, weak and sad
A cruel sun stares down`)}
p.renderHtml()
s := string(p.Html)
r := `<h1>Sun</h1>
<p>Silver leaves shine bright
They droop, boneless, weak and sad
A cruel sun stares down</p>
`
assert.Equal(t, r, string(p.Html))
}
func TestPageHtmlHashtag(t *testing.T) {
p := &Page{Body: []byte(`# Comet
Stars flicker above
Too faint to focus, so far
I am cold, alone
#Haiku #Cold_Poets`)}
p.renderHtml()
r := `<h1>Comet</h1>
<p>Stars flicker above
Too faint to focus, so far
I am cold, alone</p>
<p><a href="/search?q=%23Haiku" rel="nofollow">#Haiku</a> <a href="/search?q=%23Cold_Poets" rel="nofollow">#Cold Poets</a></p>
`
assert.Equal(t, r, string(p.Html))
if s != r {
t.Logf("The HTML is wrong: %s", s)
t.Fail()
}
}
func TestPageDir(t *testing.T) {
@@ -71,14 +68,10 @@ A slow shuffle in the dark
Moonlight floods the aisle`)}
p.save()
o, err := loadPage("testdata/moon")
assert.NoError(t, err, "load page")
assert.Equal(t, p.Body, o.Body)
assert.FileExists(t, "testdata/moon.md")
// Saving an empty page deletes it.
p = &Page{Name: "testdata/moon", Body: []byte("")}
p.save()
assert.NoFileExists(t, "testdata/moon.md")
if err != nil || string(o.Body) != string(p.Body) {
t.Logf("File in subdirectory not loaded: %s", p.Name)
t.Fail()
}
t.Cleanup(func() {
_ = os.RemoveAll("testdata")
})

View File

@@ -18,7 +18,10 @@ func score(q string, s string) int {
score += len(m)
}
}
for _, v := range strings.Fields(q) {
for _, v := range strings.Split(q, " ") {
if len(v) == 0 {
continue
}
re, err := regexp.Compile(`(?is)(\pL?)(` + v + `)(\pL?)`)
if err != nil {
continue

173
search.go
View File

@@ -1,17 +1,21 @@
package main
import (
"fmt"
trigram "github.com/dgryski/go-trigram"
"log"
"io/fs"
"path/filepath"
"slices"
"strings"
"sync"
"unicode"
"unicode/utf8"
)
// idx is the global full text search index mapping tokens to docids
var idx index
// documents is an index of document names, i.e. it maps docids to file names in our case
var docs map[docid]string
// Search is a struct containing the result of a search. Query is the
// query string and Items is the array of pages with the result.
// Currently there is no pagination of results! When a page is part of
@@ -22,23 +26,6 @@ type Search struct {
Results bool
}
// idx contains the two maps used for search. Make sure to lock and
// unlock as appropriate.
var idx = struct {
sync.RWMutex
// index is a struct containing the trigram index for search. It is
// generated at startup and updated after every page edit. The index
// is case-insensitive.
index trigram.Index
// documents is a map, mapping document ids of the index to page
// names.
documents map[trigram.DocID]string
}{}
// indexAdd reads a file and adds it to the index. This must happen
// while the idx is locked, which is true when called from loadIndex.
func indexAdd(path string, info fs.FileInfo, err error) error {
if err != nil {
return err
@@ -52,123 +39,93 @@ func indexAdd(path string, info fs.FileInfo, err error) error {
if err != nil {
return err
}
id := idx.index.Add(strings.ToLower(string(p.Body)))
idx.documents[id] = p.Name
id := idx.add(string(p.Body))
docs[id] = p.Name
return nil
}
// loadIndex loads all the pages and indexes them. This takes a while.
// It returns the number of pages indexed.
func loadIndex() (int, error) {
idx.Lock()
defer idx.Unlock()
idx.index = make(trigram.Index)
idx.documents = make(map[trigram.DocID]string)
func loadIndex() error {
idx = make(index)
docs = make(map[docid]string)
err := filepath.Walk(".", indexAdd)
if err != nil {
idx.index = nil
idx.documents = nil
return 0, err
log.Print("Indexing failed")
idx = nil
docs = nil
}
n := len(idx.documents)
return n, nil
return err
}
// updateIndex updates the index for a single page. The old text is
// loaded from the disk and removed from the index first, if it
// exists.
// updateIndex deletes the old page from the index before adding the
// new page, if an old page exists.
func (p *Page) updateIndex() {
idx.Lock()
defer idx.Unlock()
var id trigram.DocID
// This function does not rely on files actually existing, so
// let's quickly find the document id.
for docId, name := range idx.documents {
var id docid
for docId, name := range docs {
if name == p.Name {
id = docId
break
}
}
if id == 0 {
id = idx.index.Add(strings.ToLower(string(p.Body)))
idx.documents[id] = p.Name
id = idx.add(string(p.Body))
docs[id] = p.Name
} else {
o, err := loadPage(p.Name)
if err == nil {
idx.index.Delete(strings.ToLower(string(o.Body)), id)
idx.delete(string(o.Body), id)
}
idx.index.Insert(strings.ToLower(string(p.Body)), id)
id = idx.add(string(p.Body))
docs[id] = p.Name
}
}
func sortItems(a, b Page) int {
// Sort by score
if a.Score < b.Score {
return 1
} else if a.Score > b.Score {
return -1
}
// If the score is the same and both page names start
// with a number (like an ISO date), sort descending.
ra, _ := utf8.DecodeRuneInString(a.Title)
rb, _ := utf8.DecodeRuneInString(b.Title)
if unicode.IsNumber(ra) && unicode.IsNumber(rb) {
if a.Title < b.Title {
return 1
} else if a.Title > b.Title {
return -1
} else {
return 0
}
}
// Otherwise sort ascending.
if a.Title < b.Title {
return -1
} else if a.Title > b.Title {
return 1
} else {
return 0
}
}
// loadAndSummarize loads the pages named and summarizes them for the
// query give.
func loadAndSummarize(names []string, q string) []Page {
// Load and summarize the items.
items := make([]Page, len(names))
for i, name := range names {
p, err := loadPage(name)
if err != nil {
fmt.Printf("Error loading %s\n", name)
} else {
p.summarize(q)
items[i] = *p
}
}
return items
}
// search returns a sorted []Page where each page contains an extract
// of the actual Page.Body in its Page.Html.
func search(q string) []Page {
if len(q) == 0 {
return make([]Page, 0)
}
words := strings.Fields(strings.ToLower(q))
var trigrams []trigram.T
for _, word := range words {
trigrams = trigram.Extract(word, trigrams)
}
// Keep the read lock for a short as possible. Make a list of
// the names we need to load and summarize.
idx.RLock()
ids := idx.index.QueryTrigrams(trigrams)
names := make([]string, len(ids))
ids := idx.search(q)
items := make([]Page, len(ids))
for i, id := range ids {
names[i] = idx.documents[id]
name := docs[id]
p, err := loadPage(name)
if err != nil {
log.Printf("Error loading %s", name)
} else {
p.summarize(q)
items[i] = *p
}
}
idx.RUnlock()
items := loadAndSummarize(names, q)
slices.SortFunc(items, sortItems)
fn := func(a, b Page) int {
// Sort by score
if a.Score < b.Score {
return 1
} else if a.Score > b.Score {
return -1
}
// If the score is the same and both page names start
// with a number (like an ISO date), sort descending.
ra, _ := utf8.DecodeRuneInString(a.Title)
rb, _ := utf8.DecodeRuneInString(b.Title)
if unicode.IsNumber(ra) && unicode.IsNumber(rb) {
if a.Title < b.Title {
return 1
} else if a.Title > b.Title {
return -1
} else {
return 0
}
}
// Otherwise sort ascending.
if a.Title < b.Title {
return -1
} else if a.Title > b.Title {
return 1
} else {
return 0
}
}
slices.SortFunc(items, fn)
return items
}

View File

@@ -1,41 +1,40 @@
package main
import (
"github.com/stretchr/testify/assert"
"os"
"strings"
"testing"
)
var name string = "test"
// TestIndex relies on README.md being indexed
func TestIndex(t *testing.T) {
func TestSearch(t *testing.T) {
_ = os.Remove(name + ".md")
loadIndex()
q := "Oddµ"
pages := search(q)
assert.NotZero(t, len(pages))
for _, p := range pages {
assert.NotContains(t, p.Title, "<b>")
assert.True(t, strings.Contains(string(p.Body), q) || strings.Contains(string(p.Title), q))
assert.NotZero(t, p.Score)
if len(pages) == 0 {
t.Log("Search found no result")
t.Fail()
}
for _, p := range pages {
if strings.Contains(p.Title, "<b>") {
t.Logf("Page %s contains <b>", p.Name)
t.Fail()
}
if !strings.Contains(string(p.Body), q) && !strings.Contains(string(p.Title), q) {
t.Logf("Page %s does not contain %s", p.Name, q)
t.Fail()
}
if p.Score == 0 {
t.Logf("Page %s has no score", p.Name)
t.Fail()
}
}
}
func TestSearchHashtag(t *testing.T) {
loadIndex()
q := "#Another_Tag"
pages := search(q)
assert.NotZero(t, len(pages))
}
func TestIndexUpdates(t *testing.T) {
name := "test"
_ = os.Remove(name + ".md")
loadIndex()
p := &Page{Name: name, Body: []byte("This is a test.")}
p.save()
// Find the phrase
pages := search("This is a test")
pages = search("This is a test")
found := false
for _, p := range pages {
if p.Name == name {
@@ -43,9 +42,10 @@ func TestIndexUpdates(t *testing.T) {
break
}
}
assert.True(t, found)
// Find the phrase, case insensitive
if !found {
t.Logf("Page '%s' was not found", name)
t.Fail()
}
pages = search("this is a test")
found = false
for _, p := range pages {
@@ -54,9 +54,10 @@ func TestIndexUpdates(t *testing.T) {
break
}
}
assert.True(t, found)
// Find some words
if !found {
t.Logf("Page '%s' was not found using the lower case text", name)
t.Fail()
}
pages = search("this test")
found = false
for _, p := range pages {
@@ -65,9 +66,10 @@ func TestIndexUpdates(t *testing.T) {
break
}
}
assert.True(t, found)
// Update the page and no longer find it with the old phrase
if !found {
t.Logf("Page '%s' was not found using a query missing some words", name)
t.Fail()
}
p = &Page{Name: name, Body: []byte("Guvf vf n grfg.")}
p.save()
pages = search("This is a test")
@@ -78,9 +80,10 @@ func TestIndexUpdates(t *testing.T) {
break
}
}
assert.False(t, found)
// Find page using a new word
if found {
t.Logf("Page '%s' was still found using the old content: %s", name, p.Body)
t.Fail()
}
pages = search("Guvf")
found = false
for _, p := range pages {
@@ -89,8 +92,10 @@ func TestIndexUpdates(t *testing.T) {
break
}
}
assert.True(t, found)
if !found {
t.Logf("Page '%s' not found using the new content: %s", name, p.Body)
t.Fail()
}
t.Cleanup(func() {
_ = os.Remove(name + ".md")
})

View File

@@ -2,17 +2,17 @@ package main
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestSnippets(t *testing.T) {
s := `We are immersed in a sea of dead people. All the dead that have gone before us, silent now, just staring, gaping. As we move and talk and fret, never once stopping to ask ourselves or them! what it was all about. Instead we drown ourselves in noise. Incessantly we babble, surrounded by false friends claiming that all is well. And look at us! Yes, we are well. Patting our backs and expecting a pat and we do! we smugly do enjoy.`
h := `We are immersed in a sea of dead people. <b>All</b> the dead that have gone before us, silent now, just … to ask ourselves or them! what it was <b>all</b> about. Instead we drown ourselves in no<b>is</b>e. … surrounded by false friends claiming that <b>all</b> <b>is</b> <b>well</b>. And look at us! Yes, we are <b>well</b>. …`
h1 := `We are immersed in a sea of dead people. <b>All</b> the dead that have gone before us, silent now, just … to ask ourselves or them! what it was <b>all</b> about. Instead we drown ourselves in no<b>is</b>e. … surrounded by false friends claiming that <b>all</b> <b>is</b> <b>well</b>. And look at us! Yes, we are <b>well</b>. …`
q := "all is well"
r := snippets(q, s)
if r != h {
t.Logf("The snippets are wrong in 「%s」", r)
t.Fail()
}
assert.Equal(t, h1, snippets("all is well", s))
h2 := `We are immersed in a sea of dead people. All the dead that have gone before us, silent now, just … And look at us! Yes, we are well. Patting our backs and expecting a pat and we do! we smugly do <b>enjoy</b>.`
assert.Equal(t, h2, snippets("enjoy", s))
}

41
tokenizer.go Normal file
View File

@@ -0,0 +1,41 @@
// Copyright 2020 Artem Krylysov
// Copyright 2023 Alex Schroeder
//
// Licensed under the Apache License, Version 2.0 (the "License"); you
// may not use this file except in compliance with the License. You
// may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
//
// This code was originally copied from
// https://github.com/akrylysov/simplefts
// https://artem.krylysov.com/blog/2020/07/28/lets-build-a-full-text-search-engine/
//
// This file no longer does stemming and stop words.
package main
import (
"strings"
"unicode"
)
// tokenize returns a slice of tokens for the given text.
func tokenize(text string) []string {
return strings.FieldsFunc(text, func(r rune) bool {
// Split on any character that is not a letter or a number.
return !unicode.IsLetter(r) && !unicode.IsNumber(r)
})
}
// analyze analyzes the text and returns a slice of tokens.
func analyze(text string) []string {
tokens := tokenize(text)
tokens = lowercaseFilter(tokens)
return tokens
}

52
tokenizer_test.go Normal file
View File

@@ -0,0 +1,52 @@
// Copyright 2020 Artem Krylysov
// Copyright 2023 Alex Schroeder
//
// Licensed under the Apache License, Version 2.0 (the "License"); you
// may not use this file except in compliance with the License. You
// may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
// implied. See the License for the specific language governing
// permissions and limitations under the License.
//
// This code was originally copied from
// https://github.com/akrylysov/simplefts
// https://artem.krylysov.com/blog/2020/07/28/lets-build-a-full-text-search-engine/
//
// This file no longer does stemming and stop words.
package main
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestTokenizer(t *testing.T) {
testCases := []struct {
text string
tokens []string
}{
{
text: "",
tokens: []string{},
},
{
text: "a",
tokens: []string{"a"},
},
{
text: "small wild,cat!",
tokens: []string{"small", "wild", "cat"},
},
}
for _, tc := range testCases {
t.Run(tc.text, func(st *testing.T) {
assert.EqualValues(st, tc.tokens, tokenize(tc.text))
})
}
}

View File

@@ -1,22 +0,0 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="format-detection" content="telephone=no">
<meta name="viewport" content="width=device-width">
<title>Upload File</title>
<style>
html { max-width: 70ch; padding: 2ch; margin: auto; color: #111; background: #ffe; }
form, textarea { width: 100%; }
</style>
</head>
<body>
<h1>Upload File</h1>
<form action="/save" method="POST" enctype="multipart/form-data">
<input type="text" name="name" placeholder="image.jpg" autofocus required>
<p><input type="file" name="file" required>
<p><input type="submit" value="Save">
<a href="/view/index"><button type="button">Cancel</button></a></p>
</form>
</body>
</html>

View File

@@ -18,8 +18,7 @@ img { max-width: 100%; }
<header>
<a href="#main">Skip navigation</a>
<a href="/view/index">Home</a>
<a href="/edit/{{.Name}}">Edit</a>
<a href="/add/{{.Name}}">Add</a>
<a href="/edit/{{.Name}}">Edit this page</a>
<form role="search" action="/search" method="GET">
<input type="text" spellcheck="false" name="q" required>
<button>Search</button>

176
wiki.go
View File

@@ -3,23 +3,20 @@ package main
import (
"fmt"
"html/template"
"io"
"net/http"
"os"
"path/filepath"
"regexp"
"strings"
)
// Templates are parsed at startup.
var templates = template.Must(
template.ParseFiles("edit.html", "add.html", "view.html",
"search.html", "upload.html"))
var templates = template.Must(template.ParseFiles("edit.html", "view.html", "search.html"))
// validPath is a regular expression where the second group matches a
// page, so when the editHandler is called, a URL path of "/edit/foo"
// results in the editHandler being called with title "foo". The
// regular expression doesn't define the handlers (this happens in the
// main function).
// page, so when the handler for "/edit/" is called, a URL path of
// "/edit/foo" results in the editHandler being called with title
// "foo". The regular expression doesn't define the handlers (this
// happens in the main function).
var validPath = regexp.MustCompile("^/([^/]+)/(.+)$")
// titleRegexp is a regular expression matching a level 1 header line
@@ -42,31 +39,33 @@ func rootHandler(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/view/index", http.StatusFound)
}
// viewHandler serves existing files (including markdown files with
// the .md extension). If the requested file does not exist, a page
// with the same name is loaded. This means adding the .md extension
// and using the "view.html" template to render the HTML. Both
// attempts fail, the browser is redirected to an edit page.
// viewHandler renders a text file, if the name ends in ".txt" and
// such a file exists. Otherwise, it loads the page. If this didn't
// work, the browser is redirected to an edit page. Otherwise, the
// "view.html" template is used to show the rendered HTML.
func viewHandler(w http.ResponseWriter, r *http.Request, name string) {
body, err := os.ReadFile(name)
if err == nil {
w.Write(body)
return
// Short cut for text files
if strings.HasSuffix(name, ".txt") {
body, err := os.ReadFile(name)
if err == nil {
w.Write(body)
return
}
}
// Attempt to load Markdown page; edit it if this fails
p, err := loadPage(name)
if err == nil {
p.handleTitle(true)
p.renderHtml()
renderTemplate(w, "view", p)
if err != nil {
http.Redirect(w, r, "/edit/"+name, http.StatusFound)
return
}
http.Redirect(w, r, "/edit/"+name, http.StatusFound)
p.handleTitle(true)
p.renderHtml()
renderTemplate(w, "view", p)
}
// editHandler uses the "edit.html" template to present an edit page.
// When editing, the page title is not overriden by a title in the
// text. Instead, the page name is used. The edit is saved using the
// saveHandler.
// text. Instead, the page name is used.
func editHandler(w http.ResponseWriter, r *http.Request, name string) {
p, err := loadPage(name)
if err != nil {
@@ -90,83 +89,6 @@ func saveHandler(w http.ResponseWriter, r *http.Request, name string) {
http.Redirect(w, r, "/view/"+name, http.StatusFound)
}
// addHandler uses the "add.html" template to present an empty edit
// page. What you type there is appended to the page using the
// appendHandler.
func addHandler(w http.ResponseWriter, r *http.Request, name string) {
p, err := loadPage(name)
if err != nil {
p = &Page{Title: name, Name: name}
} else {
p.handleTitle(false)
}
renderTemplate(w, "add", p)
}
// appendHandler takes the "body" form parameter and appends it. The
// browser is redirected to the page view.
func appendHandler(w http.ResponseWriter, r *http.Request, name string) {
body := r.FormValue("body")
p, err := loadPage(name)
if err != nil {
p = &Page{Title: name, Name: name, Body: []byte(body)}
} else {
p.handleTitle(false)
p.Body = append(p.Body, []byte(body)...)
}
err = p.save()
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
http.Redirect(w, r, "/view/"+name, http.StatusFound)
}
// uploadHandler uses the "upload.html" template to enable uploads.
// The file is saved using the saveUploadHandler.
func uploadHandler(w http.ResponseWriter, r *http.Request) {
renderTemplate(w, "upload", nil)
}
// saveUploadHandler takes the "name" form field and the "file" form
// file and saves the file under the given name. The browser is
// redirected to the view of that file.
func saveUploadHandler(w http.ResponseWriter, r *http.Request) {
filename := r.FormValue("name")
file, _, err := r.FormFile("file")
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
defer file.Close()
// backup an existing file with the same name
_, err = os.Stat(filename)
if err != nil {
os.Rename(filename, filename + "~")
}
// create the directory, if necessary
d := filepath.Dir(filename)
if d != "." {
err := os.MkdirAll(d, 0755)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
}
// create the new file
dst, err := os.Create(filename)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
defer dst.Close()
if _, err := io.Copy(dst, file); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
http.Redirect(w, r, "/view/"+filename, http.StatusFound)
}
// makeHandler returns a handler that uses the URL path without the
// first path element as its argument, e.g. if the URL path is
// /edit/foo/bar, the editHandler is called with "foo/bar" as its
@@ -193,6 +115,20 @@ func searchHandler(w http.ResponseWriter, r *http.Request) {
renderTemplate(w, "search", s)
}
// searchCommand prints a search result to the command line.
func searchCommand() {
q := strings.Join(os.Args[1:], " ")
fmt.Println("Indexing all pages")
loadIndex()
fmt.Printf("Searching for %s\n", q)
for _, p := range search(q) {
t := strings.ReplaceAll(string(p.Html), "<b>", "\033[31;1m")
t = strings.ReplaceAll(t, "</b>", "\033[0m")
fmt.Printf("%s.md\n" + "\033[1;4m%s\033[0m (%d)\n" + "%s\n\n",
p.Name, p.Title, p.Score, t)
}
}
// getPort returns the environment variable ODDMU_PORT or the default
// port, "8080".
func getPort() string {
@@ -203,49 +139,23 @@ func getPort() string {
return port
}
// scheduleLoadIndex calls loadIndex and prints some messages before
// and after. For testing, call loadIndex directly and skip the
// messages.
func scheduleLoadIndex() {
fmt.Print("Indexing pages\n")
n, err := loadIndex()
if err == nil {
fmt.Printf("Indexed %d pages\n", n)
} else {
fmt.Println("Indexing failed")
}
}
// scheduleLoadLanguages calls loadLanguages and prints some messages before
// and after. For testing, call loadLanguages directly and skip the
// messages.
func scheduleLoadLanguages() {
fmt.Print("Loading languages\n")
n := loadLanguages()
fmt.Printf("Loaded %d languages\n", n)
}
func serve() {
http.HandleFunc("/", rootHandler)
http.HandleFunc("/view/", makeHandler(viewHandler))
http.HandleFunc("/edit/", makeHandler(editHandler))
http.HandleFunc("/save/", makeHandler(saveHandler))
http.HandleFunc("/add/", makeHandler(addHandler))
http.HandleFunc("/append/", makeHandler(appendHandler))
http.HandleFunc("/upload", uploadHandler)
http.HandleFunc("/save", saveUploadHandler)
http.HandleFunc("/search", searchHandler)
go scheduleLoadIndex()
go scheduleLoadLanguages()
fmt.Println("Indexing all pages")
loadIndex()
port := getPort()
fmt.Printf("Serving a wiki on port %s\n", port)
http.ListenAndServe(":"+port, nil)
}
func main() {
if len(os.Args) == 1 {
serve()
if len(os.Args) > 1 {
searchCommand()
} else {
commands()
serve()
}
}

View File

@@ -1,184 +0,0 @@
package main
import (
"bytes"
"github.com/stretchr/testify/assert"
"mime/multipart"
"net/http"
"net/http/httptest"
"net/url"
"os"
"regexp"
"strings"
"testing"
)
// HTTPHeaders is a helper that returns HTTP headers of the response. It returns
// nil if building a new request fails.
func HTTPHeaders(handler http.HandlerFunc, method, url string, values url.Values, header string) []string {
w := httptest.NewRecorder()
req, err := http.NewRequest(method, url+"?"+values.Encode(), nil)
if err != nil {
return nil
}
handler(w, req)
return w.Result().Header[header]
}
// HTTPRedirectTo checks that the request results in a redirect and it
// checks the destination of the redirect. It returns whether the
// request did in fact result in a redirect. Note: This method assumes
// that POST requests ignore the query part of the URL.
func HTTPRedirectTo(t *testing.T, handler http.HandlerFunc, method, url string, values url.Values, destination string) bool {
w := httptest.NewRecorder()
var req *http.Request
var err error
if method == http.MethodPost {
body := strings.NewReader(values.Encode())
req, err = http.NewRequest(method, url, body)
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
} else {
req, err = http.NewRequest(method, url+"?"+values.Encode(), nil)
}
assert.NoError(t, err)
handler(w, req)
code := w.Code
isRedirectCode := code >= http.StatusMultipleChoices && code <= http.StatusTemporaryRedirect
assert.True(t, isRedirectCode, "Expected HTTP redirect status code for %q but received %d", url+"?"+values.Encode(), code)
headers := w.Result().Header["Location"]
assert.True(t, len(headers) == 1 && headers[0] == destination,
"Expected HTTP redirect location %s for %q but received %v", destination, url+"?"+values.Encode(), headers)
return isRedirectCode
}
// HTTPUploadAndRedirectTo checks that the request results in a redirect and it
// checks the destination of the redirect. It returns whether the
// request did in fact result in a redirect.
func HTTPUploadAndRedirectTo(t *testing.T, handler http.HandlerFunc, url, contentType string, body *bytes.Buffer, destination string) bool {
w := httptest.NewRecorder()
req, err := http.NewRequest("POST", url, body)
req.Header.Set("Content-Type", contentType)
assert.NoError(t, err)
handler(w, req)
code := w.Code
isRedirectCode := code >= http.StatusMultipleChoices && code <= http.StatusTemporaryRedirect
assert.True(t, isRedirectCode, "Expected HTTP redirect status code for %q but received %d", url, code)
headers := w.Result().Header["Location"]
assert.True(t, len(headers) == 1 && headers[0] == destination,
"Expected HTTP redirect location %s for %q but received %v", destination, url, headers)
return isRedirectCode
}
func TestRootHandler(t *testing.T) {
HTTPRedirectTo(t, rootHandler, "GET", "/", nil, "/view/index")
}
// relies on index.md in the current directory!
func TestViewHandler(t *testing.T) {
assert.Regexp(t, regexp.MustCompile("Welcome to Oddµ"),
assert.HTTPBody(makeHandler(viewHandler), "GET", "/view/index", nil))
}
// wipes testdata
func TestEditSave(t *testing.T) {
_ = os.RemoveAll("testdata")
data := url.Values{}
data.Set("body", "Hallo!")
HTTPRedirectTo(t, makeHandler(viewHandler), "GET", "/view/testdata/alex", nil, "/edit/testdata/alex")
assert.HTTPStatusCode(t, makeHandler(editHandler), "GET", "/edit/testdata/alex", nil, 200)
HTTPRedirectTo(t, makeHandler(saveHandler), "POST", "/save/testdata/alex", data, "/view/testdata/alex")
assert.Regexp(t, regexp.MustCompile("Hallo!"),
assert.HTTPBody(makeHandler(viewHandler), "GET", "/view/testdata/alex", nil))
t.Cleanup(func() {
_ = os.RemoveAll("testdata")
})
}
// wipes testdata
func TestAddAppend(t *testing.T) {
_ = os.RemoveAll("testdata")
p := &Page{Name: "testdata/fire", Body: []byte(`# Fire
Orange sky above
Reflects a distant fire
It's not `)}
p.save()
data := url.Values{}
data.Set("body", "barbecue")
assert.Regexp(t, regexp.MustCompile("a distant fire"),
assert.HTTPBody(makeHandler(viewHandler), "GET", "/view/testdata/fire", nil))
assert.NotRegexp(t, regexp.MustCompile("a distant fire"),
assert.HTTPBody(makeHandler(addHandler), "GET", "/add/testdata/fire", nil))
HTTPRedirectTo(t, makeHandler(appendHandler), "POST", "/append/testdata/fire", data, "/view/testdata/fire")
assert.Regexp(t, regexp.MustCompile("Its not barbecue"),
assert.HTTPBody(makeHandler(viewHandler), "GET", "/view/testdata/fire", nil))
t.Cleanup(func() {
_ = os.RemoveAll("testdata")
})
}
// wipes testdata
func TestUpload(t *testing.T) {
_ = os.RemoveAll("testdata")
assert.HTTPStatusCode(t, uploadHandler, "GET", "/upload", nil, 200)
form := new(bytes.Buffer)
writer := multipart.NewWriter(form)
field, err := writer.CreateFormField("name")
assert.NoError(t, err)
_, err = field.Write([]byte("testdata/ok.txt"))
assert.NoError(t, err)
file, err := writer.CreateFormFile("file", "example.txt");
assert.NoError(t, err)
file.Write([]byte("Hello!"))
err = writer.Close()
assert.NoError(t, err)
t.Log(writer.FormDataContentType())
HTTPUploadAndRedirectTo(t, saveUploadHandler, "/upload", writer.FormDataContentType(), form, "/view/testdata/ok.txt")
assert.Regexp(t, regexp.MustCompile("Hello!"),
assert.HTTPBody(makeHandler(viewHandler), "GET", "/view/testdata/ok.txt", nil))
t.Cleanup(func() {
_ = os.RemoveAll("testdata")
})
}
// wipes testdata
func TestPageTitleWithAmp(t *testing.T) {
_ = os.RemoveAll("testdata")
p := &Page{Name: "testdata/Rock & Roll", Body: []byte("Dancing")}
p.save()
assert.Regexp(t, regexp.MustCompile("Rock &amp; Roll"),
assert.HTTPBody(makeHandler(viewHandler), "GET", "/view/testdata/Rock%20%26%20Roll", nil))
p = &Page{Name: "testdata/Rock & Roll", Body: []byte("# Sex & Drugs & Rock'n'Roll\nOh no!")}
p.save()
assert.Regexp(t, regexp.MustCompile("Sex &amp; Drugs"),
assert.HTTPBody(makeHandler(viewHandler), "GET", "/view/testdata/Rock%20%26%20Roll", nil))
t.Cleanup(func() {
_ = os.RemoveAll("testdata")
})
}
func TestPageTitleWithQuestionMark(t *testing.T) {
_ = os.RemoveAll("testdata")
p := &Page{Name: "testdata/How about no?", Body: []byte("No means no")}
p.save()
body := assert.HTTPBody(makeHandler(viewHandler), "GET", "/view/testdata/How%20about%20no%3F", nil)
assert.Contains(t, body, "No means no")
assert.Contains(t, body, "<a href=\"/edit/testdata/How%20about%20no%3F\">Edit</a>")
t.Cleanup(func() {
_ = os.RemoveAll("testdata")
})
}