Rebasely Beta: data may be purged

examplesstorefront Public

Search the catalog with D1 full text search #4

  1. Justin Torre opened this

    Shoppers can only browse the catalog by name today. This adds GET /search with D1 full text search, in a second D1 database so search load stays away from checkout.

    Why a second database

    A D1 database runs one query at a time. Search is going to be the busiest read we have, because the search box asks on every pause in typing and each request ranks matches with bm25(). In the storefront database those queries would wait in the same line as checkout's batches. The index is also derived data that we will retune and rebuild, and that is easier when it is not next to the orders. Decision 0011 has the full reasoning and supersedes 0001 ("one D1 database for the shop's data"); the shop's data itself stays in one database.

    What changed

    • storefront-search, a new D1 database bound as SEARCH, with an FTS5 table products_fts (search/migrations/0001_products_fts.sql). The tokenizer is porter unicode61 remove_diacritics 2, so mugs finds mug and creme finds Crème.
    • GET /search?q=&limit= in src/search.ts. toMatch() quotes every word and makes the last one a prefix, so FTS5 syntax in the input is just text and results follow someone who is still typing. Names weigh ten times more than descriptions. The index returns ids only; the products are read back from the catalog with a new activeProducts() in src/catalog.ts, so price and stock are always current and retired products never show up.
    • Keeping it in step. PUT /admin/products/:id updates the product's row in the index after saving it, or removes it when the product is inactive. If the index write fails, the answer is 503 search_index_failed and the same request can be sent again. POST /admin/search/rebuild refills the index from the catalog.
    • Architecture: a new search component, storefront-api's new edge to it, search added to the catalog view, and a new flow, "Searching the catalog", from the merchant's edit to the shopper's results. The Visual tab shows all three.
    • Knowledge base: a new search concept, decision 0011 (0001 marked superseded), and updates to the catalog, testing and local development notes and the README.

    Rollout

    1. npx wrangler d1 create storefront-search and put its id in wrangler.jsonc.
    2. npx wrangler d1 migrations apply storefront-search --remote.
    3. Deploy storefront-api, then POST /admin/search/rebuild once to index the current catalog.

    Testing

    • test/search.test.ts (9 tests) runs the real FTS5 queries on Node's SQLite: name before description, prefixes, stemming and accents, current stock after a checkout, edits and retired products, limit and bad queries, the rebuild and its admin check, and a failed index write.
    • npm test (58 tests) and npm run typecheck pass. kb check and arch check pass.
  2. src/search.ts line 55
    55  const remove = search.prepare("DELETE FROM products_fts WHERE product_id = ?").bind(p.id);
    1. Justin Torre

      product_id is UNINDEXED, so this delete scans the whole FTS table on every product write. At our few hundred products that is nothing. If the catalog ever gets into the tens of thousands, we should give each product a fixed rowid in the index and delete by rowid instead.

    1. Justin Torre

      Tried it against the seeded catalog with wrangler dev: after the rebuild, 'linen' returns the apron first and the tote second, 'cand' finds the beeswax candle while typing, and retiring the apron drops it from the next search. The Visual tab shows the new search database next to db in the catalog view, and the flow reads well end to end.

    1. Justin Torre

      One correction to my last comment: with the seed data only the apron matches 'linen'. The tote coming second is from the test fixture, where its description mentions the apron. Retiring the apron and 'mugs' finding the enamel mug both check out in wrangler dev.

  3. Justin Torre pushed to catalog-search b748afa → 75e565d

Open

Ready to merge

  • Failedduragit/ci/testout of memoryDetails
  • No conflictsmain can fast-forward to the head

Sign in to comment.