Search

beachcomber is the SummerCMS counterpart of Laravel Scout, as WinterCMS applications use it without a queue. A model opts in by implementing beachcomber.Searchable; after a write of such a model commits, its row is reloaded and its document written to, or removed from, the search index. A search asks the index for matching IDs, and the application loads the rows from the database.

Engines#

search.driver selects the engine. The default, null, indexes nothing and finds nothing. The typesense engine, from the beachcomber/typesense package, talks to a Typesense server over its HTTP API. An engine registers itself from its package's init function, so the application imports the engine package for its side effect; an unknown name stops the start-up. search.prefix is prepended to every index name, for example to keep staging and production apart on one server:

cfg, err := compass.Open(compass.Options{Dir: "config", Env: "development", Environ: []string{}})
if err != nil {
	fmt.Println(err)
	return
}
_ = cfg.Set("search.prefix", "staging_")
svc, err := beachcomber.From(backpack.New(cfg)) // search.driver defaults to null
if err != nil {
	fmt.Println(err)
	return
}
fmt.Println(svc.Engine().Name(), svc.Engine().Configured(), svc.IndexName(&Post{}))
ids, err := svc.Engine().SearchIDs(context.Background(), svc.IndexName(&Post{}), beachcomber.Query{Q: "go"})
fmt.Println(ids, err)

_ = cfg.Set("search.driver", "elastic")
_, err = beachcomber.From(backpack.New(cfg))
fmt.Println(err != nil)
// Output:
// null false staging_acme_blog_posts
// [] <nil>
// true

An engine implements beachcomber.Engine and registers with beachcomber.RegisterEngine. The examples on this page use a small in-memory engine that matches titles:

func (e *memoryEngine) SearchIDs(ctx context.Context, index string, q beachcomber.Query) ([]string, error) {
	e.mu.Lock()
	defer e.mu.Unlock()
	ids := []string{}
	for id, d := range e.docs[index] {
		if strings.Contains(strings.ToLower(fmt.Sprint(d["title"])), strings.ToLower(q.Q)) {
			ids = append(ids, id)
		}
	}
	slices.Sort(ids)
	return ids, nil
}

Searchable models#

A model implements beachcomber.Searchable with three methods, and needs no import of beachcomber to do so:

// SearchableAs is the index name, before search.prefix.
func (Post) SearchableAs() string { return "acme_blog_posts" }

// ShouldBeSearchable keeps drafts out of the index.
func (p *Post) ShouldBeSearchable() bool { return p.Published }

// ToSearchableArray builds the document from the committed row.
func (p *Post) ToSearchableArray(ctx context.Context, db *gorm.DB) (map[string]any, error) {
	return map[string]any{
		"id":      strconv.FormatUint(uint64(p.ID), 10),
		"blog_id": int64(p.BlogID),
		"title":   p.Title,
	}, nil
}

ToSearchableArray runs after the commit on a fresh copy of the row, so it may query related rows. A row whose ShouldBeSearchable is false, a soft-deleted row and a deleted row have their documents removed. A model can also implement beachcomber.IndexSchemaProvider, the schema the engine creates a missing index with, and beachcomber.SearchKeyer, a document key other than the primary key.

Sync after commit#

beachcomber.From installs GORM callbacks that register the sync with lagoon.AfterCommit:

• Inside lagoon.Transaction, the sync runs after the commit, and never after a rollback.
• A single-statement write syncs after GORM commits it.
• Inside a plain GORM transaction the sync is skipped with a warning, because the commit cannot be observed. Wrap such writes in lagoon.Transaction, or call beachcomber.Service.Sync after the commit.

The sync runs inline in the writing goroutine, so a search right after a save finds the document, and it is bounded by the engine's timeout. It is never fatal: a failure is logged as search: sync failed with the index, key and operation, never the document or the API key, and the write stays committed. A write without a primary key value, such as Model(&Post{}).Where(...).Updates(...), cannot be synced row by row; bulk paths call beachcomber.Service.Sync and beachcomber.Service.Remove per row, or reindex.

The kill-switch#

Nothing is sent when the engine is not configured (the null engine, or Typesense without an API key), when no database is published, or when the application's beachcomber.Gate reports off. Install the gate from a plugin's Boot. A gate must treat a read error as off:

svc.SetGate(beachcomber.GateFunc(func(ctx context.Context, db *gorm.DB) bool {
	var enabled bool
	err := db.WithContext(ctx).Raw(`SELECT search_enabled FROM acme_blog_settings WHERE id = 1`).Scan(&enabled).Error
	return err == nil && enabled
}))

Searching#

beachcomber.Engine.SearchIDs returns the IDs of matching documents, in the engine's order. They are candidates, not answers: the index can be stale (a write it missed, a document from before a permission change) and its filters are only as good as the document. Re-check every ID in SQL, with the same ownership, visibility and soft-delete conditions the rest of the API applies, before a row reaches a response:

ids, err := svc.Engine().SearchIDs(ctx, svc.IndexName(&Post{}), beachcomber.Query{
	Q:        term,
	QueryBy:  []string{"title"},
	FilterBy: "blog_id:=" + strconv.FormatUint(uint64(blogID), 10),
})
if err != nil {
	return nil, err
}
// The ids are candidates from an index that may be stale or loosely
// filtered: re-check every one in SQL before exposing a row.
var posts []Post
err = db.WithContext(ctx).
	Where("id IN ? AND blog_id = ? AND published", ids, blogID).
	Order("id").
	Find(&posts).Error
return posts, err

An empty result is an empty list, never an error.

Typesense#

The Typesense engine follows the Scout Typesense wire contract, so indexes built by a WinterCMS application can be searched by the port:

driver: typesense
typesense:
  host: 127.0.0.1
  port: 8108
  protocol: http

in config/search.yaml, with the key in SUMMER_SEARCH__TYPESENSE__API_KEY. Without an API key nothing is ever sent. A search is one request to the collection's search endpoint, and the engine returns the hit IDs in Typesense's order:

// A stand-in Typesense node.
node := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
	q := r.URL.Query()
	fmt.Println(r.Method, r.URL.Path, "key sent:", r.Header.Get("X-TYPESENSE-API-KEY") != "")
	fmt.Println("q", q.Get("q"), "query_by", q.Get("query_by"), "filter_by", q.Get("filter_by"))
	fmt.Fprint(w, `{"hits":[{"document":{"id":"3"}},{"document":{"id":"1"}}]}`)
}))
defer node.Close()
u, _ := url.Parse(node.URL)
port, _ := strconv.Atoi(u.Port())

engine := typesense.New(typesense.Config{
	APIKey:            "test-only-key", // SUMMER_SEARCH__TYPESENSE__API_KEY
	Host:              u.Hostname(),
	Port:              port,
	Protocol:          "http",
	ConnectionTimeout: 2 * time.Second,
})
ids, err := engine.SearchIDs(context.Background(), "acme_blog_posts", beachcomber.Query{
	Q:        "go",
	QueryBy:  []string{"title"},
	FilterBy: "blog_id:=7",
})
fmt.Println(ids, err)

// Without an API key the engine is not configured and nothing is sent.
fmt.Println(typesense.New(typesense.Config{}).Configured())
// Output:
// GET /collections/acme_blog_posts/documents/search key sent: true
// q go query_by title filter_by blog_id:=7
// [3 1] <nil>
// false

Each request times out after search.typesense.connection_timeout_seconds (2 seconds by default). A failed answer is a typesense.StatusError with the method, path and status, never the answer body.