examplesstorefront Public
Search the catalog with D1 full text search #4
- src/search.ts line 55
55 const remove = search.prepare("DELETE FROM products_fts WHERE product_id = ?").bind(p.id);-
Justin Torre
product_id is UNINDEXED, so this delete scans the whole FTS table on every product write. At our few hundred products that is nothing. If the catalog ever gets into the tens of thousands, we should give each product a fixed rowid in the index and delete by rowid instead.
-
-
Justin Torre
Tried it against the seeded catalog with wrangler dev: after the rebuild, 'linen' returns the apron first and the tote second, 'cand' finds the beeswax candle while typing, and retiring the apron drops it from the next search. The Visual tab shows the new search database next to db in the catalog view, and the flow reads well end to end.
-
-
Justin Torre
One correction to my last comment: with the seed data only the apron matches 'linen'. The tote coming second is from the test fixture, where its description mentions the apron. Retiring the apron and 'mugs' finding the enamel mug both check out in wrangler dev.
-
Ready to merge
- Failedduragit/ci/testout of memoryDetails
- No conflictsmain can fast-forward to the head
Sign in to comment.
Shoppers can only browse the catalog by name today. This adds
GET /searchwith D1 full text search, in a second D1 database so search load stays away from checkout.Why a second database
A D1 database runs one query at a time. Search is going to be the busiest read we have, because the search box asks on every pause in typing and each request ranks matches with
bm25(). In thestorefrontdatabase those queries would wait in the same line as checkout's batches. The index is also derived data that we will retune and rebuild, and that is easier when it is not next to the orders. Decision 0011 has the full reasoning and supersedes 0001 ("one D1 database for the shop's data"); the shop's data itself stays in one database.What changed
storefront-search, a new D1 database bound asSEARCH, with an FTS5 tableproducts_fts(search/migrations/0001_products_fts.sql). The tokenizer isporter unicode61 remove_diacritics 2, somugsfindsmugandcremefindsCrème.GET /search?q=&limit=insrc/search.ts.toMatch()quotes every word and makes the last one a prefix, so FTS5 syntax in the input is just text and results follow someone who is still typing. Names weigh ten times more than descriptions. The index returns ids only; the products are read back from the catalog with a newactiveProducts()insrc/catalog.ts, so price and stock are always current and retired products never show up.PUT /admin/products/:idupdates the product's row in the index after saving it, or removes it when the product is inactive. If the index write fails, the answer is503 search_index_failedand the same request can be sent again.POST /admin/search/rebuildrefills the index from the catalog.searchcomponent, storefront-api's new edge to it,searchadded to the catalog view, and a new flow, "Searching the catalog", from the merchant's edit to the shopper's results. The Visual tab shows all three.searchconcept, decision 0011 (0001 marked superseded), and updates to the catalog, testing and local development notes and the README.Rollout
npx wrangler d1 create storefront-searchand put its id inwrangler.jsonc.npx wrangler d1 migrations apply storefront-search --remote.POST /admin/search/rebuildonce to index the current catalog.Testing
test/search.test.ts(9 tests) runs the real FTS5 queries on Node's SQLite: name before description, prefixes, stemming and accents, current stock after a checkout, edits and retired products,limitand bad queries, the rebuild and its admin check, and a failed index write.npm test(58 tests) andnpm run typecheckpass.kb checkandarch checkpass.