Search results are based on page titles only

Add pagination of the results
Sorting and scoring still requires the loading of all the found pages, but the summary is only computed for the pages on the page.
2023-09-16 23:36:17 +02:00 · 2023-09-16 16:14:59 +02:00 · 2023-09-16 13:26:06 +02:00 · 2023-09-16 09:59:30 +02:00 · 2023-09-15 21:37:33 +02:00 · 2023-09-15 16:03:52 +02:00
23 changed files with 399 additions and 116 deletions
--- a/README.md
+++ b/README.md
@@ -31,10 +31,25 @@ Oddmu. 🙃

 This wiki uses a [Markdown
 library](https://github.com/gomarkdown/markdown) to generate the web
-pages from Markdown. There is no additional wiki markup. Most
-importantly, double square brackets are not a link. If you're used to
-that, it'll be strange as you need to repeat the name: `[like
-this](like this)`.
+pages from Markdown. There are two extensions Oddmu adds to the
+library: local links and hashtags.
+
+Local links use double square brackets `[[like this]]`. If you need to
+change the link text, you need to use regular Markdown. Don't forget
+to [percent-encode](https://en.wikipedia.org/wiki/Percent-encoding)
+the link target. Example: `[here](like%20this)`.
+
+Hashtags link to searches for the hashtag. Hashtags are separate from
+titles because there is no space after the hash. Use the underscore to
+use hashtags consisting of multiple words.
+
+```
+# Title
+
+Text
+
+#Tag #Another_Tag
+```

 The Markdown processor comes with a few extensions, some of which are
 enable by default:
@@ -71,19 +86,6 @@ Internet
 : Vector of transmission for pictures of cats
 ```

-There is another extension made: hashtags link to searches for the
-hashtag. Hashtags are separate from titles because there is no space
-after the hash. Use the underscore to use hashtags consisting of
-multiple words.
-
-```
-# Title
-
-Text
-
-#Tag #Another_Tag
-```
-
 ## Templates

 The template files are the HTML files in the working directory:
@@ -117,7 +119,13 @@ is a byte array and that's why we need to call `printf`).

 For the `search.html` template only:

-`{{.Results}}` indicates if there were any search results.
+`{{.Previous}}`, `{{.Page}}`, `{{.Next}}` and `{{.Last}}` are the
+previous, current, next and last page number in the results since
+doing arithmetics in templates is hard. The first page number is 1.
+
+`{{.More}}` indicates if there are any more search results.
+
+`{{.Results}}` indicates if there were any search results at all.

 `{{.Items}}` is an array of pages, each containing a search result. A
 search result is a page (with the properties seen above). Thus, to
@@ -431,6 +439,22 @@ and "rail", a search for "mail" returns a match because the trigrams
 "mai" and "ail" are found. In this situation, the result has a score
 of 0.

+The sorting of all the pages, however, does not depend on scoring!
+Computing the score is expensive because the page must be loaded from
+disk. Therefore, results are sorted by title:
+
+- If the page title contains the query string, it gets sorted first.
+- If the page title begins with a number, it is sorted descending.
+- All other pages follow, sorted ascending.
+
+The effect is that first, the pages with matches in the page title are
+shown, and then all the others. Within these two groups, the most
+recent blog posts are shown first, if and only if the page title
+begins with an ISO date like 2023-09-16.
+
+The score and highlighting of snippets is used to help visitors decide
+which links to click.
+
 ## Limitations

 Page titles are filenames with `.md` appended. If your filesystem
--- a/add.html
+++ b/add.html
@@ -1,5 +1,5 @@
 <!DOCTYPE html>
-<html>
+<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="format-detection" content="telephone=no">
@@ -13,7 +13,7 @@ form, textarea { width: 100%; }
  <body>
    <h1>Adding to {{.Title}}</h1>
    <form action="/append/{{.Name}}" method="POST">
-      <textarea name="body" rows="20" cols="80" placeholder="Text" autofocus required></textarea>
+      <textarea name="body" rows="20" cols="80" placeholder="Text" lang="" autofocus required></textarea>
      <p><input type="submit" value="Add">
        <a href="/view/{{.Name}}"><button type="button">Cancel</button></a></p>
    </form>
--- a/add_append_test.go
+++ b/add_append_test.go
@@ -12,7 +12,7 @@ import (
 func TestAddAppend(t *testing.T) {
 	_ = os.RemoveAll("testdata")
 	index.load()
-	
+
 	p := &Page{Name: "testdata/fire", Body: []byte(`# Fire
 Orange sky above
 Reflects a distant fire
--- a/commands.go
+++ b/commands.go
@@ -7,21 +7,24 @@ import (

 func commands() {
 	if len(os.Args) == 3 && os.Args[1] == "html" {
-		p, err := loadPage(os.Args[2]);
+		p, err := loadPage(os.Args[2])
 		if err != nil {
-			fmt.Println(err);
+			fmt.Println(err)
 		} else {
-			p.renderHtml();
-			fmt.Println(p.Html);
+			p.renderHtml()
+			fmt.Println(p.Html)
 		}
 	} else if len(os.Args) > 2 && os.Args[1] == "search" {
 		index.load()
 		for _, q := range os.Args[2:] {
-			items := search(q)
+			items, more, _ := search(q, 1)
 			fmt.Printf("Search %s: %d results\n", q, len(items))
 			for _, p := range items {
 				fmt.Printf("* %s (%d)\n", p.Title, p.Score)
 			}
+			if more {
+				fmt.Printf("There are more results\n")
+			}
 		}
 	} else {
 		fmt.Printf("Unknown command: %v\n", os.Args[1:])
--- a/concurrency_test.go
+++ b/concurrency_test.go
@@ -10,6 +10,6 @@ func TestLoadAndSearch(t *testing.T) {
 	index.reset()
 	go index.load()
 	q := "Oddµ"
-	pages := search(q)
+	pages, _, _ := search(q, 1)
 	assert.Zero(t, len(pages))
 }
--- a/edit.html
+++ b/edit.html
@@ -1,5 +1,5 @@
 <!DOCTYPE html>
-<html>
+<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="format-detection" content="telephone=no">
@@ -15,7 +15,7 @@ form, textarea { width: 100%; }
    <form action="/save/{{.Name}}" method="POST">
      <textarea name="body" rows="20" cols="80" placeholder="# Title

-Text" autofocus>{{printf "%s" .Body}}</textarea>
+Text" lang="" autofocus>{{printf "%s" .Body}}</textarea>
      <p><input type="submit" value="Save">
        <a href="/view/{{.Name}}"><button type="button">Cancel</button></a></p>
    </form>
--- a/go.mod
+++ b/go.mod
@@ -11,12 +11,14 @@ require (
 )

 require (
+	github.com/anthonynsimon/bild v0.13.0 // indirect
 	github.com/aymerick/douceur v0.2.0 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
 	github.com/gorilla/css v1.0.0 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
 	github.com/shopspring/decimal v1.3.1 // indirect
 	golang.org/x/exp v0.0.0-20221106115401-f9659909a136 // indirect
+	golang.org/x/image v0.0.0-20190703141733-d6a02ce849c9 // indirect
 	golang.org/x/net v0.12.0 // indirect
 	google.golang.org/protobuf v1.31.0 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
--- a/go.sum
+++ b/go.sum
@@ -1,9 +1,18 @@
+github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
+github.com/anthonynsimon/bild v0.13.0 h1:mN3tMaNds1wBWi1BrJq0ipDBhpkooYfu7ZFSMhXt1C8=
+github.com/anthonynsimon/bild v0.13.0/go.mod h1:tpzzp0aYkAsMi1zmfhimaDyX1xjn2OUc1AJZK/TF0AE=
+github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
 github.com/aymerick/douceur v0.2.0 h1:Mv+mAeH1Q+n9Fr+oyamOlAkUNPWPlA8PPGR0QAaYuPk=
 github.com/aymerick/douceur v0.2.0/go.mod h1:wlT5vV2O3h55X9m7iVYN0TBM0NH/MmbLnd30/FjWUq4=
+github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
+github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk=
+github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
+github.com/cpuguy83/go-md2man v1.0.10/go.mod h1:SmD6nW6nTyfqj6ABTjUi3V3JVMnlJmwcJI5acqYI6dE=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/dgryski/go-trigram v0.0.0-20160407183937-79ec494e1ad0 h1:b+7JSiBM+hnLQjP/lXztks5hnLt1PS46hktG9VOJgzo=
 github.com/dgryski/go-trigram v0.0.0-20160407183937-79ec494e1ad0/go.mod h1:qzKC/DpcxK67zaSHdCmIv3L9WJViHVinYXN2S7l3RM8=
+github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
 github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
 github.com/gomarkdown/markdown v0.0.0-20230716120725-531d2d74bc12 h1:uK3X/2mt4tbSGoHvbLBHUny7CKiuwUip3MArtukol4E=
 github.com/gomarkdown/markdown v0.0.0-20230716120725-531d2d74bc12/go.mod h1:JDGcbDT52eL4fju3sZ4TeHGsQwhG9nbDV21aMyhwPoA=
@@ -14,25 +23,47 @@ github.com/google/go-cmp v0.5.8 h1:e6P7q2lk1O+qJJb4BtCQXlK8vWEO8V1ZeuEdJNOqZyg=
 github.com/google/go-cmp v0.5.8/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
 github.com/gorilla/css v1.0.0 h1:BQqNyPTi50JCFMTw/b67hByjMVXZRwGha6wxVGkeihY=
 github.com/gorilla/css v1.0.0/go.mod h1:Dn721qIggHpt4+EFCcTLTU/vk5ySda2ReITrtgBl60c=
+github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
+github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
+github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
 github.com/microcosm-cc/bluemonday v1.0.25 h1:4NEwSfiJ+Wva0VxN5B8OwMicaJvD8r9tlJWm9rtloEg=
 github.com/microcosm-cc/bluemonday v1.0.25/go.mod h1:ZIOjCQp1OrzBBPIJmfX4qDYFuhU02nx4bn030ixfHLE=
+github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
+github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
+github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
 github.com/pemistahl/lingua-go v1.4.0 h1:ifYhthrlW7iO4icdubwlduYnmwU37V1sbNrwhKBR4rM=
 github.com/pemistahl/lingua-go v1.4.0/go.mod h1:ECuM1Hp/3hvyh7k8aWSqNCPlTxLemFZsRjocUf3KgME=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
 github.com/shopspring/decimal v1.3.1 h1:2Usl1nmF/WZucqkFZhnfFYxxxu8LG21F6nPQBE5gKV8=
 github.com/shopspring/decimal v1.3.1/go.mod h1:DKyhrW/HYNuLGql+MJL6WCR6knT2jwCFRcu2hWCYk4o=
+github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
+github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
+github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU=
+github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=
+github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
+github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s=
+github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
 github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
 github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
+github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0=
+github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
+golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
 golang.org/x/exp v0.0.0-20221106115401-f9659909a136 h1:Fq7F/w7MAa1KJ5bt2aJ62ihqp9HDcRuyILskkpIAurw=
 golang.org/x/exp v0.0.0-20221106115401-f9659909a136/go.mod h1:CxIveKay+FTh1D0yPZemJVgC/95VzuuOLq5Qi4xnoYc=
+golang.org/x/image v0.0.0-20190703141733-d6a02ce849c9 h1:uc17S921SPw5F2gJo7slQ3aqvr2RwpL7eb3+DZncu3s=
+golang.org/x/image v0.0.0-20190703141733-d6a02ce849c9/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
 golang.org/x/net v0.12.0 h1:cfawfvKITfUsFCeJIHJrbSxpeu/E81khclypR0GVT50=
 golang.org/x/net v0.12.0/go.mod h1:zEVYFnQC7m/vmpQFELhcD1EWkZlX69l4oqgmer6hfKA=
+golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
 google.golang.org/protobuf v1.31.0 h1:g0LDEJHgrBl9N9r17Ru3sqWhkIx2NB67okBHPwC7hs8=
 google.golang.org/protobuf v1.31.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
 gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
--- a/index.go
+++ b/index.go
@@ -21,6 +21,9 @@ type Index struct {
 	// documents is a map, mapping document ids of the index to
 	// page names.
 	documents map[trigram.DocID]string
+
+	// names is a map, mapping page names to titles.
+	titles map[string]string
 }

 // idx is the global Index per wiki.
@@ -30,6 +33,7 @@ var index Index
 func (idx *Index) reset() {
 	idx.index = nil
 	idx.documents = nil
+	idx.titles = nil
 }

 // add reads a file and adds it to the index. This must happen while
@@ -47,8 +51,10 @@ func (idx *Index) add(path string, info fs.FileInfo, err error) error {
 	if err != nil {
 		return err
 	}
+	p.handleTitle(false)
 	id := idx.index.Add(strings.ToLower(string(p.Body)))
 	idx.documents[id] = p.Name
+	idx.titles[p.Name] = p.Title
 	return nil
 }

@@ -59,6 +65,7 @@ func (idx *Index) load() (int, error) {
 	defer idx.Unlock()
 	idx.index = make(trigram.Index)
 	idx.documents = make(map[trigram.DocID]string)
+	idx.titles = make(map[string]string)
 	err := filepath.Walk(".", idx.add)
 	if err != nil {
 		idx.reset()
@@ -90,23 +97,27 @@ func (p *Page) updateIndex() {
 		o, err := loadPage(p.Name)
 		if err == nil {
 			index.index.Delete(strings.ToLower(string(o.Body)), id)
+			o.handleTitle(false)
+			delete(index.titles, o.Title)
 		}
 		index.index.Insert(strings.ToLower(string(p.Body)), id)
+		p.handleTitle(false)
+		index.titles[p.Name] = p.Title
 	}
 }

+// searchDocuments searches the index for a string. This requires the
+// index to be locked.
 func searchDocuments(q string) []string {
-		words := strings.Fields(strings.ToLower(q))
+	words := strings.Fields(strings.ToLower(q))
 	var trigrams []trigram.T
 	for _, word := range words {
 		trigrams = trigram.Extract(word, trigrams)
 	}
-	index.RLock()
 	ids := index.index.QueryTrigrams(trigrams)
 	names := make([]string, len(ids))
 	for i, id := range ids {
 		names[i] = index.documents[id]
 	}
-	index.RUnlock()
 	return names
 }
--- a/index_test.go
+++ b/index_test.go
@@ -11,19 +11,19 @@ import (
 func TestIndex(t *testing.T) {
 	index.load()
 	q := "Oddµ"
-	pages := search(q)
+	pages, _, _ := search(q, 1)
 	assert.NotZero(t, len(pages))
 	for _, p := range pages {
 		assert.NotContains(t, p.Title, "<b>")
 		assert.True(t, strings.Contains(string(p.Body), q) || strings.Contains(string(p.Title), q))
-		assert.NotZero(t, p.Score)
+		assert.NotZero(t, p.Score, "Score %d for %s", p.Score, p.Name)
 	}
 }

 func TestSearchHashtag(t *testing.T) {
 	index.load()
 	q := "#Another_Tag"
-	pages := search(q)
+	pages, _, _ := search(q, 1)
 	assert.NotZero(t, len(pages))
 }

@@ -35,7 +35,7 @@ func TestIndexUpdates(t *testing.T) {
 	p.save()

 	// Find the phrase
-	pages := search("This is a test")
+	pages, _, _ := search("This is a test", 1)
 	found := false
 	for _, p := range pages {
 		if p.Name == name {
@@ -46,7 +46,7 @@ func TestIndexUpdates(t *testing.T) {
 	assert.True(t, found)

 	// Find the phrase, case insensitive
-	pages = search("this is a test")
+	pages, _, _ = search("this is a test", 1)
 	found = false
 	for _, p := range pages {
 		if p.Name == name {
@@ -57,7 +57,7 @@ func TestIndexUpdates(t *testing.T) {
 	assert.True(t, found)

 	// Find some words
-	pages = search("this test")
+	pages, _, _ = search("this test", 1)
 	found = false
 	for _, p := range pages {
 		if p.Name == name {
@@ -70,7 +70,7 @@ func TestIndexUpdates(t *testing.T) {
 	// Update the page and no longer find it with the old phrase
 	p = &Page{Name: name, Body: []byte("Guvf vf n grfg.")}
 	p.save()
-	pages = search("This is a test")
+	pages, _, _ = search("This is a test", 1)
 	found = false
 	for _, p := range pages {
 		if p.Name == name {
@@ -81,7 +81,7 @@ func TestIndexUpdates(t *testing.T) {
 	assert.False(t, found)

 	// Find page using a new word
-	pages = search("Guvf")
+	pages, _, _ = search("Guvf", 1)
 	found = false
 	for _, p := range pages {
 		if p.Name == name {
--- a/page.go
+++ b/page.go
@@ -59,7 +59,7 @@ func (p *Page) save() error {
 	filename := p.Name + ".md"
 	s := bytes.ReplaceAll(p.Body, []byte{'\r'}, []byte{})
 	if len(s) == 0 {
-		_ = os.Rename(filename, filename + "~")
+		_ = os.Rename(filename, filename+"~")
 		return os.Remove(filename)
 	}
 	p.Body = s
@@ -72,7 +72,7 @@ func (p *Page) save() error {
 			return err
 		}
 	}
-	_ = os.Rename(filename, filename + "~")
+	_ = os.Rename(filename, filename+"~")
 	return os.WriteFile(filename, s, 0644)
 }

@@ -104,6 +104,30 @@ func (p *Page) handleTitle(replace bool) {
 	}
 }

+// wikiLink returns an inline parser function. This indirection is
+// required because we want to call the previous definition in case
+// this is not a wikiLink.
+func wikiLink(p *parser.Parser,	fn func(p *parser.Parser, data []byte, offset int) (int, ast.Node)) func(p *parser.Parser, data []byte, offset int) (int, ast.Node) {
+	return func (p *parser.Parser, original []byte, offset int) (int, ast.Node) {
+		data := original[offset:]
+		n := len(data)
+		// minimum: [[X]]
+		if n < 5 || data[1] != '[' {
+			return fn(p, original, offset)
+		}
+		i := 2
+		for i+1 < n && data[i] != ']' && data[i+1] != ']' {
+			i++
+		}
+		text := data[2:i+1]
+		link := &ast.Link{
+			Destination: []byte(url.PathEscape(string(text))),
+		}
+		ast.AppendChild(link, &ast.Text{Leaf: ast.Leaf{Literal: text}})
+		return i+3, link
+	}
+}
+
 func hashtag(p *parser.Parser, data []byte, offset int) (int, ast.Node) {
 	data = data[offset:]
 	i := 0
@@ -126,6 +150,8 @@ func hashtag(p *parser.Parser, data []byte, offset int) (int, ast.Node) {
 // renderHtml renders the Page.Body to HTML and sets Page.Html.
 func (p *Page) renderHtml() {
 	parser := parser.New()
+	prev := parser.RegisterInline('[', nil)
+	parser.RegisterInline('[', wikiLink(parser, prev))
 	parser.RegisterInline('#', hashtag)
 	maybeUnsafeHTML := markdown.ToHTML(p.Body, parser, nil)
 	p.Name = nameEscape(p.Name)
@@ -160,10 +186,14 @@ func (p *Page) plainText() string {
 	return string(text)
 }

-// summarize for query string q sets Page.Html to an extract.
-func (p *Page) summarize(q string) {
+// score sets Page.Title and computes Page.Score.
+func (p *Page) score(q string) {
 	p.handleTitle(true)
 	p.Score = score(q, string(p.Body)) + score(q, p.Title)
+}
+
+// summarize sets Page.Html to an extract and sets Page.Language.
+func (p *Page) summarize(q string) {
 	t := p.plainText()
 	p.Html = sanitize(snippets(q, t))
 	p.Language = language(t)
--- a/page_test.go
+++ b/page_test.go
@@ -62,6 +62,22 @@ I am cold, alone</p>
 	assert.Equal(t, r, string(p.Html))
 }

+func TestPageHtmlWikiLink(t *testing.T) {
+	p := &Page{Body: []byte(`# Photos and Books
+Blue and green and black
+Sky and grass and [ragged cliffs](cliffs)
+Our [[time together]]
+`)}
+	p.renderHtml()
+	r := `<h1>Photos and Books</h1>
+
+<p>Blue and green and black
+Sky and grass and <a href="cliffs" rel="nofollow">ragged cliffs</a>
+Our <a href="time%20together" rel="nofollow">time together</a></p>
+`
+	assert.Equal(t, r, string(p.Html))
+}
+
 // wipes testdata
 func TestPageDir(t *testing.T) {
 	_ = os.RemoveAll("testdata")
@@ -71,7 +87,7 @@ From bed to bathroom
 A slow shuffle in the dark
 Moonlight floods the aisle`)}
 	p.save()
-	
+
 	o, err := loadPage("testdata/moon")
 	assert.NoError(t, err, "load page")
 	assert.Equal(t, p.Body, o.Body)
@@ -84,7 +100,7 @@ Moonlight floods the aisle`)}

 	// But the backup still exists.
 	assert.FileExists(t, "testdata/moon.md~")
-	
+
 	t.Cleanup(func() {
 		_ = os.RemoveAll("testdata")
 	})
--- a/score_test.go
+++ b/score_test.go
@@ -94,7 +94,7 @@ func TestScorePageAndMarkup(t *testing.T) {
 	s := `The Transjovian Council accepts new members. If you think we'd be a good fit, apply for an account. Contact [Alex Schroeder](https://alexschroeder.ch/wiki/Contact). Mail is best. Encrypted mail is best. [Delta Chat](https://delta.chat/de/) is a messenger app that uses encrypted mail. It's the bestest best.`
 	p := &Page{Title: "Test", Name: "Test", Body: []byte(s)}
 	q := "wiki"
-	p.summarize(q)
+	p.score(q)
 	// "wiki" is not visible in the plain text but the score is no affected:
 	// - wiki, all, whole, beginning, end (5)
 	if p.Score != 5 {
--- a/search.go
+++ b/search.go
@@ -4,6 +4,8 @@ import (
 	"fmt"
 	"net/http"
 	"slices"
+	"strconv"
+	"strings"
 	"unicode"
 	"unicode/utf8"
 )
@@ -14,67 +16,97 @@ import (
 // a search result, Body and Html are simple extracts.
 type Search struct {
 	Query   string
-	Items   []Page
+	Items   []*Page
+	Previous int
+	Page    int
+	Next    int
+	Last    int
+	More    bool
 	Results bool
 }

-func sortItems(a, b Page) int {
-	// Sort by score
-	if a.Score < b.Score {
-		return 1
-	} else if a.Score > b.Score {
-		return -1
-	}
-	// If the score is the same and both page names start
-	// with a number (like an ISO date), sort descending.
-	ra, _ := utf8.DecodeRuneInString(a.Title)
-	rb, _ := utf8.DecodeRuneInString(b.Title)
-	if unicode.IsNumber(ra) && unicode.IsNumber(rb) {
-		if a.Title < b.Title {
-			return 1
-		} else if a.Title > b.Title {
+// sortNames returns a sort function that sorts in three stages: 1.
+// whether the query string matches the page title; 2. descending if
+// the page titles start with a digit; 3. otherwise ascending.
+// Access to the index requires a read lock!
+func sortNames(q string) func (a, b string) int {
+	return func (a, b string) int {
+		// If only one page contains the query string, it
+		// takes precedence.
+		ia := strings.Contains(index.titles[a], q)
+		ib := strings.Contains(index.titles[b], q)
+		if (ia && !ib) {
 			return -1
+		} else if (!ia && ib) {
+			return 1
+		}
+		// If both page names start with a number (like an ISO date),
+		// sort descending.
+		ra, _ := utf8.DecodeRuneInString(a)
+		rb, _ := utf8.DecodeRuneInString(b)
+		if unicode.IsNumber(ra) && unicode.IsNumber(rb) {
+			if a < b {
+				return 1
+			} else if a > b {
+				return -1
+			} else {
+				return 0
+			}
+		}
+		// Otherwise sort ascending.
+		if a < b {
+			return -1
+		} else if a > b {
+			return 1
 		} else {
 			return 0
 		}
 	}
-	// Otherwise sort ascending.
-	if a.Title < b.Title {
-		return -1
-	} else if a.Title > b.Title {
-		return 1
-	} else {
-		return 0
-	}
 }

-// loadAndSummarize loads the pages named and summarizes them for the
-// query give.
-func loadAndSummarize(names []string, q string) []Page {
-	// Load and summarize the items.
-	items := make([]Page, len(names))
+// load the pages named.
+func load(names []string) []*Page {
+	items := make([]*Page, len(names))
 	for i, name := range names {
 		p, err := loadPage(name)
 		if err != nil {
 			fmt.Printf("Error loading %s\n", name)
 		} else {
-			p.summarize(q)
-			items[i] = *p
+			items[i] = p
 		}
 	}
 	return items
 }

+// itemsPerPage says how many items to print on a page of search
+// results.
+const itemsPerPage = 20
+
 // search returns a sorted []Page where each page contains an extract
-// of the actual Page.Body in its Page.Html.
-func search(q string) []Page {
+// of the actual Page.Body in its Page.Html. Page size is 20. The
+// boolean return value indicates whether there are more results.
+func search(q string, page int) ([]*Page, bool, int) {
 	if len(q) == 0 {
-		return make([]Page, 0)
+		return make([]*Page, 0), false, 0
 	}
+	index.RLock()
 	names := searchDocuments(q)
-	items := loadAndSummarize(names, q)
-	slices.SortFunc(items, sortItems)
-	return items
+	slices.SortFunc(names, sortNames(q))
+	index.RUnlock()
+	from := itemsPerPage*(page-1)
+	if from > len(names) {
+		return make([]*Page, 0), false, 0
+	}
+	to := from + itemsPerPage
+	if to > len(names) {
+		to = len(names)
+	}
+	items := load(names[from:to])
+	for _, p := range items {
+		p.score(q)
+		p.summarize(q)
+	}
+	return items, to < len(names), len(names)/itemsPerPage+1
 }

 // searchHandler presents a search result. It uses the query string in
@@ -82,7 +114,12 @@ func search(q string) []Page {
 // page found, the HTML is just an extract of the actual body.
 func searchHandler(w http.ResponseWriter, r *http.Request) {
 	q := r.FormValue("q")
-	items := search(q)
-	s := &Search{Query: q, Items: items, Results: len(items) > 0}
+	page, err := strconv.Atoi(r.FormValue("page"))
+	if err != nil {
+		page = 1
+	}
+	items, more, last := search(q, page)
+	s := &Search{Query: q, Items: items, Previous: page-1, Page: page, Next: page+1, Last: last,
+		Results: len(items) > 0, More: more}
 	renderTemplate(w, "search", s)
 }
--- a/search.html
+++ b/search.html
@@ -1,5 +1,5 @@
 <!DOCTYPE html>
-<html>
+<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="format-detection" content="telephone=no">
@@ -10,23 +10,32 @@ html { max-width: 65ch; padding: 2ch; margin: auto; color: #111; background: #ff
 body { hyphens: auto; }
 header a { margin-right: 1ch; }
 form { display: inline-block; }
+input#search { width: 20ch; }
+button { background: #eee; color: inherit; border-radius: 4px; border-width: 1px; }
 img { max-width: 20%; }
 .result { font-size: larger }
 .score { font-size: smaller; opacity: 0.8; }
    </style>
  </head>
-  <body lang="en">
+  <body>
    <header>
      <a href="#main">Skip navigation</a>
      <a href="/view/index">Home</a>
      <form role="search" action="/search" method="GET">
-        <input type="text" value="{{.Query}}" spellcheck="false" name="q" required>
-        <button>Search</button>
+        <label for="search">Search:</label>
+        <input id="search" type="text" value="{{.Query}}" spellcheck="false" name="q" required>
+        <button>Go</button>
      </form>
    </header>
    <main id="main">
      <h1>Search for {{.Query}}</h1>
      {{if .Results}}
+      <p>
+        {{if gt .Page 2}}<a href="/search?q={{.Query}}&page=1">First</a>{{end}}
+        {{if gt .Page 1}}<a href="/search?q={{.Query}}&page={{.Previous}}">Previous</a>{{end}}
+        Page {{.Page}}
+        {{if .More}}<a href="/search?q={{.Query}}&page={{.Next}}">Next</a>{{end}}
+        {{if lt .Next .Last}}<a href="/search?q={{.Query}}&page={{.Last}}">Last</a>{{end}}
      {{range .Items}}
      <article lang="{{.Language}}">
        <p><a class="result" href="/view/{{.Name}}">{{.Title}}</a>
--- a/search_test.go
+++ b/search_test.go
@@ -2,8 +2,8 @@ package main

 import (
 	"github.com/stretchr/testify/assert"
-	"testing"
 	"net/url"
+	"testing"
 )

 func TestSearch(t *testing.T) {
--- a/snippets.go
+++ b/snippets.go
@@ -75,7 +75,7 @@ func snippets(q string, s string) string {
 				to = len(s)
 			}
 			end := strings.LastIndex(s[:to], " ")
-			if end == -1 || end <= j + wl {
+			if end == -1 || end <= j+wl {
 				// OK, look for a longer word
 				end = strings.Index(s[to:], " ")
 				if end == -1 {
--- a/snippets_test.go
+++ b/snippets_test.go
@@ -41,5 +41,5 @@ func TestSnippetsLong(t *testing.T) {
 	assert.Equal(t,
 		"VWwXetig mty8fORN UNia4NFm SQsfyFHk BLDdgVnc AcvKP2fs q8KxPH1A IaCzFj96 J0S2fqca jp3ElV9f ULIZ1aMX … Auk5RBWs tMpfMMlU p6jGYq3Z rTIBTHVM zGFwFwQi <b>j4O1AY21</b> BJnaiScY",
 		snippets("j4O1AY21", s))
-	
+
 }
--- a/upload.html
+++ b/upload.html
@@ -7,14 +7,32 @@
    <title>Upload File</title>
    <style>
 html { max-width: 70ch; padding: 2ch; margin: auto; color: #111; background: #ffe; }
+body { hyphens: auto; }
 form, textarea { width: 100%; }
+label { display: inline-block; width: 20ch }
    </style>
  </head>
-  <body>
+  <body lang="en">
    <h1>Upload File</h1>
    <form action="/drop/{{.}}" method="POST" enctype="multipart/form-data">
-      <input type="text" name="name" placeholder="image.jpg" autofocus required>
-      <p><input type="file" name="file" required>
+      <p>When uploading pictures from a phone, its filename is going to be something cryptic like IMG_1234.JPG.
+        Please provide your own filename.
+      <p><label for="text">Filename to use:</label>
+        <input id="text" name="name" type="text" placeholder="image.jpg" autofocus required>
+      <p>If the uploaded file is a picture from a phone, it is going to be too big for your site.
+        Sadly, resizing only works for JPG and PNG files. Luckily, most pictures from a phone camera are JPG images.
+        Feel free to specify a max width of 1200 pixels, for example.
+      <p><label for="maxwidth">Max width:</label>
+        <input id="maxwidth" name="maxwidth" type="number" min="10" placeholder="1200">
+      <p>If the uploaded file is a JPEG-encoded picture, like most pictures from a phone, you can specify a quality.
+        Typically, a quality of 60 is not too bad and a quality of 90 is more than enough.
+      <p><label for="quality">Quality:</label>
+        <input id="quality" name="quality" type="number" min="1" max="99" placeholder="75">
+      <p>Finally, pick the file or photo to upload.
+        Picture metadata is only removed if the picture gets resized.
+        Providing a new max width is recommended for all pictures.
+      <p><label for="file">Pick file to upload:</label>
+        <input type="file" name="file" required>
      <p><input type="submit" value="Save">
        <a href="/view/index"><button type="button">Cancel</button></a></p>
    </form>
--- a/upload_drop.go
+++ b/upload_drop.go
@@ -1,10 +1,16 @@
 package main

 import (
+	"github.com/anthonynsimon/bild/imgio"
+	"github.com/anthonynsimon/bild/transform"
+	"image/jpeg"
 	"io"
 	"net/http"
 	"os"
 	"path"
+	"path/filepath"
+	"strconv"
+	"strings"
 )

 // uploadHandler uses the "upload.html" template to enable uploads.
@@ -28,7 +34,12 @@ func dropHandler(w http.ResponseWriter, r *http.Request, dir string) {
 		http.Error(w, "file exists", http.StatusInternalServerError)
 		return
 	}
-	filename := r.FormValue("name")
+	name := r.FormValue("name")
+	filename := filepath.Base(name)
+	if filename == "." || filepath.Dir(name) != "." {
+		http.Error(w, "no filename", http.StatusInternalServerError)
+		return
+	}
 	file, _, err := r.FormFile("file")
 	if err != nil {
 		http.Error(w, err.Error(), http.StatusInternalServerError)
@@ -38,19 +49,64 @@ func dropHandler(w http.ResponseWriter, r *http.Request, dir string) {
 	// backup an existing file with the same name
 	_, err = os.Stat(filename)
 	if err != nil {
-		os.Rename(filename, filename + "~")
+		os.Rename(filename, filename+"~")
 	}
 	// create the new file
-	dst, err := os.Create(d + "/" + filename)
+	path := d + "/" + filename
+	dst, err := os.Create(path)
 	if err != nil {
-            http.Error(w, err.Error(), http.StatusInternalServerError)
-            return
-        }
+		http.Error(w, err.Error(), http.StatusInternalServerError)
+		return
+	}
 	defer dst.Close()
 	if _, err := io.Copy(dst, file); err != nil {
-            http.Error(w, err.Error(), http.StatusInternalServerError)
-            return
-        }
-	http.Redirect(w, r, "/view/"+filename, http.StatusFound)
-}
+		http.Error(w, err.Error(), http.StatusInternalServerError)
+		return
+	}
+	// if a resize was requested
+	maxwidth := r.FormValue("maxwidth")
+	if len(maxwidth) > 0 {
+		mw, err := strconv.Atoi(maxwidth)
+		if err != nil {
+			http.Error(w, err.Error(), http.StatusInternalServerError)
+			return
+		}
+		ext := strings.ToLower(filepath.Ext(path))
+		var encoder imgio.Encoder
+		switch ext {
+		case ".png":
+			encoder = imgio.PNGEncoder()
+		case ".jpg", ".jpeg":
+			q := jpeg.DefaultQuality
+			quality := r.FormValue("quality")
+			if len(quality) > 0 {
+				q, err = strconv.Atoi(quality)
+				if err != nil {
+					http.Error(w, err.Error(), http.StatusInternalServerError)
+					return
+				}
+			}
+			encoder = imgio.JPEGEncoder(q)
+		default:
+			http.Error(w, "only .png, .jpg, or .jpeg files are supported", http.StatusInternalServerError)
+			return
+		}
+		img, err := imgio.Open(path)
+		if err != nil {
+			http.Error(w, err.Error(), http.StatusInternalServerError)
+			return
+		}
+		rect := img.Bounds()
+		width := rect.Max.X - rect.Min.X
+		if width > mw {
+			height := (rect.Max.Y - rect.Min.Y) * mw / width
+			img = transform.Resize(img, mw, height, transform.Linear)
+			if err := imgio.Save(path, img, encoder); err != nil {
+				http.Error(w, err.Error(), http.StatusInternalServerError)
+				return
+			}
+		}

+	}
+	http.Redirect(w, r, "/view/"+path, http.StatusFound)
+}
--- a/upload_drop_test.go
+++ b/upload_drop_test.go
@@ -3,6 +3,9 @@ package main
 import (
 	"bytes"
 	"github.com/stretchr/testify/assert"
+	"image"
+	"image/jpeg"
+	"image/png"
 	"mime/multipart"
 	"os"
 	"regexp"
@@ -14,19 +17,19 @@ func TestUpload(t *testing.T) {
 	_ = os.RemoveAll("testdata")
 	// for uploads, the directory is not created automatically
 	os.MkdirAll("testdata", 0755)
-	assert.HTTPStatusCode(t, makeHandler(uploadHandler, false), "GET", "/upload/", nil, 200)
+	assert.HTTPStatusCode(t, makeHandler(uploadHandler, false), "GET", "/upload/testdata/", nil, 200)
 	form := new(bytes.Buffer)
 	writer := multipart.NewWriter(form)
 	field, err := writer.CreateFormField("name")
 	assert.NoError(t, err)
-	_, err = field.Write([]byte("testdata/ok.txt"))
+	_, err = field.Write([]byte("ok.txt"))
 	assert.NoError(t, err)
-	file, err := writer.CreateFormFile("file", "example.txt");
+	file, err := writer.CreateFormFile("file", "example.txt")
 	assert.NoError(t, err)
 	file.Write([]byte("Hello!"))
 	err = writer.Close()
 	assert.NoError(t, err)
-	HTTPUploadAndRedirectTo(t, makeHandler(dropHandler, false), "/drop/",
+	HTTPUploadAndRedirectTo(t, makeHandler(dropHandler, false), "/drop/testdata/",
 		writer.FormDataContentType(), form, "/view/testdata/ok.txt")
 	assert.Regexp(t, regexp.MustCompile("Hello!"),
 		assert.HTTPBody(makeHandler(viewHandler, true), "GET", "/view/testdata/ok.txt", nil))
@@ -34,3 +37,43 @@ func TestUpload(t *testing.T) {
 		_ = os.RemoveAll("testdata")
 	})
 }
+
+// wipes testdata
+func TestUploadPng(t *testing.T) {
+	_ = os.RemoveAll("testdata")
+	// for uploads, the directory is not created automatically
+	os.MkdirAll("testdata", 0755)
+	form := new(bytes.Buffer)
+	writer := multipart.NewWriter(form)
+	field, _ := writer.CreateFormField("name")
+	field.Write([]byte("ok.png"))
+	file, _ := writer.CreateFormFile("file", "ok.png")
+	img := image.NewRGBA(image.Rect(0, 0, 20, 20))
+	png.Encode(file, img)
+	writer.Close()
+	HTTPUploadAndRedirectTo(t, makeHandler(dropHandler, false), "/drop/testdata/",
+		writer.FormDataContentType(), form, "/view/testdata/ok.png")
+	t.Cleanup(func() {
+		_ = os.RemoveAll("testdata")
+	})
+}
+
+// wipes testdata
+func TestUploadJpg(t *testing.T) {
+	_ = os.RemoveAll("testdata")
+	// for uploads, the directory is not created automatically
+	os.MkdirAll("testdata", 0755)
+	form := new(bytes.Buffer)
+	writer := multipart.NewWriter(form)
+	field, _ := writer.CreateFormField("name")
+	field.Write([]byte("ok.jpg"))
+	file, _ := writer.CreateFormFile("file", "ok.jpg")
+	img := image.NewRGBA(image.Rect(0, 0, 20, 20))
+	jpeg.Encode(file, img, &jpeg.Options{Quality: 90})
+	writer.Close()
+	HTTPUploadAndRedirectTo(t, makeHandler(dropHandler, false), "/drop/testdata/",
+		writer.FormDataContentType(), form, "/view/testdata/ok.jpg")
+	t.Cleanup(func() {
+		_ = os.RemoveAll("testdata")
+	})
+}
--- a/view.go
+++ b/view.go
@@ -1,6 +1,6 @@
 package main

-import(
+import (
 	"net/http"
 	"os"
 )
--- a/view.html
+++ b/view.html
@@ -1,5 +1,5 @@
 <!DOCTYPE html>
-<html>
+<html lang="{{.Language}}">
  <head>
    <meta charset="utf-8">
    <meta name="format-detection" content="telephone=no">
@@ -10,11 +10,13 @@ html { max-width: 65ch; padding: 1ch; margin: auto; color: #111; background: #ff
 body { hyphens: auto; }
 header a { margin-right: 1ch; }
 form { display: inline-block; }
+input#search { width: 12ch; }
+button { background: #eee; color: inherit; border-radius: 4px; border-width: 1px; }
 footer { border-top: 1px solid #888 }
 img { max-width: 100%; }
    </style>
  </head>
-  <body lang="{{.Language}}">
+  <body>
    <header>
      <a href="#main">Skip navigation</a>
      <a href="/view/index">Home</a>
@@ -22,8 +24,9 @@ img { max-width: 100%; }
      <a href="/add/{{.Name}}">Add</a>
      <a href="/upload/{{.Dir}}">Upload</a>
      <form role="search" action="/search" method="GET">
-        <input type="text" spellcheck="false" name="q" required>
-        <button>Search</button>
+        <label for="search">Search:</label>
+        <input id="search" type="text" spellcheck="false" name="q" required>
+        <button>Go</button>
      </form>
    </header>
    <main id="main">
Author	SHA1	Message	Date
Alex Schroeder	153a179d92	Search results are based on page titles only	2023-09-16 23:36:17 +02:00
Alex Schroeder	d9797aac75	Add pagination of the results Sorting and scoring still requires the loading of all the found pages, but the summary is only computed for the pages on the page.	2023-09-16 16:14:59 +02:00
Alex Schroeder	005500457e	Add wiki markup	2023-09-16 13:26:06 +02:00
Alex Schroeder	2635d5f852	Improve HTML template accessibility Used the web accessibility evaluation tool (WAVE) at wave.webaim.org and fixed some errors.	2023-09-16 09:59:30 +02:00
Alex Schroeder	a79f4558b6	Fix file extension check	2023-09-15 21:37:33 +02:00
Alex Schroeder	d1c2b8e27c	go fmt	2023-09-15 16:03:52 +02:00
Alex Schroeder	dd939e2c86	HTML changes to the upload template	2023-09-15 16:03:38 +02:00
Alex Schroeder	475c7071ba	Add image resizing	2023-09-15 14:13:20 +02:00