examples/storefront Public
Map the search index and record why it has its own database
1 of 1 check failedAdds the search component and the searching the catalog flow, puts search in the catalog view, and writes the search concept. Decision 0011 supersedes 0001: the shop's data stays in one database, the index moves to a second one. The catalog, testing and local development notes and the README cover the new route and the rebuild.
Changed files
11 files changed142 additions14 deletionsagainst 54fcfaa
- Added.rebasely/architecture/flows/catalog-search.yaml+17−0
- Changed.rebasely/architecture/system.yaml+13−3
- Changed.rebasely/architecture/views/catalog.yaml+2−2
- Changed.rebasely/kb/concepts/catalog.md+6−2
- Added.rebasely/kb/concepts/search.md+51−0
- Changed.rebasely/kb/decisions/0001-d1-for-shop-data.md+2−1
- Added.rebasely/kb/decisions/0011-search-index-in-its-own-database.md+31−0
- Changed.rebasely/kb/guides/local-dev.md+9−2
- Changed.rebasely/kb/guides/testing.md+3−2
- Changed.rebasely/kb/index.md+3−1
- ChangedREADME.md+5−1
Added.rebasely/architecture/flows/catalog-search.yaml+17−0
| @@ -0,0 +1,17 @@ | ||
| +title: Searching the catalog | ||
| +description: >- | ||
| + The merchant renames a product, and a shopper searches for it a moment later. Every product | ||
| + write updates the product's row in the search index, a separate D1 database. A search matches | ||
| + in the index, then reads the products from the catalog, so prices, stock and retired products | ||
| + are always current. | ||
| +actors: [merchant, shopper] | ||
| +steps: | ||
| + - { from: merchant, to: storefront-api, label: "PUT /admin/products/linen-apron, admin token" } | ||
| + - { from: storefront-api, to: db, label: upsert the product } | ||
| + - { from: storefront-api, to: search, label: "one batch: delete the product's row, insert its name and description" } | ||
| + - { from: storefront-api, to: merchant, label: "the product, or 503 search_index_failed to retry" } | ||
| + - { from: shopper, to: storefront-api, label: "GET /search?q=linen apr" } | ||
| + - { from: storefront-api, to: search, label: "MATCH '\"linen\" \"apr\"*', ranked by bm25, name weighted 10" } | ||
| + - { from: search, to: storefront-api, label: "product ids, best match first" } | ||
| + - { from: storefront-api, to: db, label: "read those products, active ones only" } | ||
| + - { from: storefront-api, to: shopper, label: "the products in match order, with current price and stock" } | ||
Changed.rebasely/architecture/system.yaml+13−3
| @@ -2,13 +2,14 @@ | ||
| storefront-api: | ||
| kind: worker | ||
| description: >- | ||
| - The public HTTP API (wrangler.jsonc, Worker storefront-api). Shoppers browse the catalog, fill | ||
| - carts and check out; the merchant edits products and uploads their images with the admin | ||
| - token. Its OrdersService entrypoint is how the other Workers change orders: it is the only | ||
| + The public HTTP API (wrangler.jsonc, Worker storefront-api). Shoppers browse and search the | ||
| + catalog, fill carts and check out; the merchant edits products and uploads their images with | ||
| + the admin token. Its OrdersService entrypoint is how the other Workers change orders: it is the only | ||
| Worker with the database binding. A cron trigger expires unpaid orders every five minutes. | ||
| owns: [src/**] | ||
| talks_to: | ||
| - { to: db, via: d1 } | ||
| + - { to: search, via: d1 } | ||
| - { to: images, via: r2 } | ||
| - { to: stripe, via: https } | ||
| - { to: order-events, via: queue } | ||
| @@ -37,6 +38,15 @@ | ||
| file in migrations/, applied with wrangler d1 migrations apply. | ||
| binding: DB | ||
| owns: [migrations/**] | ||
| + search: | ||
| + kind: d1 | ||
| + description: >- | ||
| + Database storefront-search: an FTS5 index of the names and descriptions of active products, | ||
| + for GET /search. Derived from the catalog in db, updated on every product write and refilled | ||
| + by POST /admin/search/rebuild, so it can be dropped and rebuilt at any time. Kept apart from | ||
| + db so search reads never queue behind checkout's writes. Schema in search/migrations/. | ||
| + binding: SEARCH | ||
| + owns: [search/**] | ||
| images: | ||
| kind: r2-bucket | ||
| description: >- | ||
Changed.rebasely/architecture/views/catalog.yaml+2−2
| @@ -1,2 +1,2 @@ | ||
| -description: What browsing the shop touches, and where product images come from. | ||
| -components: [storefront-api, db, images] | ||
| +description: What browsing and searching the shop touches, and where product images come from. | ||
| +components: [storefront-api, db, search, images] | ||
Changed.rebasely/kb/concepts/catalog.md+6−2
| @@ -6,7 +6,7 @@ | ||
| describes: | ||
| - src/catalog.ts | ||
| - migrations/0001_products.sql | ||
| -reviewed: 2026-10-13 | ||
| +reviewed: 2026-10-16 | ||
| --- | ||
| # Catalog | ||
| @@ -30,9 +30,13 @@ | ||
| A cursor rather than `?page=3` because the query starts right after the last row (`(name, id) > (?, ?)`) on the `products_by_name` index, so page 40 costs the same as page 1, and a product added or retired between two requests does not shift the next page by one. | ||
| +## Searching | ||
| + | ||
| +`GET /search` finds products by the words in their name and description; see [[search]]. The index lives in another database and only returns ids, so search results are read from this table with `activeProducts()` and have the same fields, prices and stock as the rest of the catalog. | ||
| + | ||
| ## Writing | ||
| -The merchant writes products with `PUT /admin/products/:id` and the admin token. The call is an upsert: the same request creates the product or replaces its fields, which makes it safe to retry and easy to drive from a spreadsheet export. Validation lives in `parseProductInput`: | ||
| +The merchant writes products with `PUT /admin/products/:id` and the admin token. The call is an upsert: the same request creates the product or replaces its fields, which makes it safe to retry and easy to drive from a spreadsheet export. It also updates the product's row in the search index. Validation lives in `parseProductInput`: | ||
| - the id is a lowercase slug of at most 64 characters, because it appears in URLs and image keys; | ||
| - `price_cents` is a whole number above zero; | ||
Added.rebasely/kb/concepts/search.md+51−0
| @@ -0,0 +1,51 @@ | ||
| +--- | ||
| +id: search | ||
| +type: concept | ||
| +owner: justin | ||
| +component: search | ||
| +describes: | ||
| + - src/search.ts | ||
| + - search/migrations/** | ||
| +reviewed: 2026-10-16 | ||
| +--- | ||
| +# Search | ||
| + | ||
| +`GET /search?q=linen apr` returns active products whose name or description match, best match first: | ||
| + | ||
| +``` | ||
| +GET /search?q=linen%20apr&limit=24 | ||
| +{ "query": "linen apr", "products": [{ "id": "linen-apron", "name": "Linen work apron", ... }] } | ||
| +``` | ||
| + | ||
| +`q` is 1 to 100 characters. `limit` works as on `GET /products`: 1 to 100, default 24. There is no paging; a shopper who does not find it in the first page types more. | ||
| + | ||
| +## The index | ||
| + | ||
| +The index is one FTS5 table, `products_fts`, in its own D1 database, `storefront-search` ([[0011-search-index-in-its-own-database]]). It has a row per active product with the product's id (stored, not searched), name and description. Nothing else about a product is in it. | ||
| + | ||
| +The tokenizer is `porter unicode61 remove_diacritics 2`: `mugs` finds `mug`, and `creme` finds `Crème`. | ||
| + | ||
| +## Queries | ||
| + | ||
| +`toMatch()` turns what the shopper typed into an FTS5 query. It keeps letters and digits, quotes each word, and makes the last one a prefix: | ||
| + | ||
| +| typed | MATCH | | ||
| +| --- | --- | | ||
| +| `linen apr` | `"linen" "apr"*` | | ||
| +| `apron" OR name:mug` | `"apron" "OR" "name" "mug"*` | | ||
| +| `***` | nothing; the answer is an empty list | | ||
| + | ||
| +Quoting means FTS5 operators and column filters in the input are only words, so no input can make the query fail. The prefix on the last word lets results follow someone who is still typing. Up to eight words are used. | ||
| + | ||
| +Matches are ranked with `bm25(products_fts, 0, 10, 1)`: a word in the name counts ten times as much as one in the description, so the apron comes before the tote whose description mentions an apron. | ||
| + | ||
| +## Reading the results | ||
| + | ||
| +The index only answers with ids. `activeProducts()` (`src/catalog.ts`) reads those products from the catalog with `json_each`, keeps the active ones and returns them in the index's order. Price and stock are therefore always the catalog's, including stock that checkout took a second ago, and a product retired since it was indexed is left out. | ||
| + | ||
| +## Keeping it in step | ||
| + | ||
| +- `PUT /admin/products/:id` saves the product, then calls `indexProduct()`, which deletes the product's row and inserts it again in one batch, or only deletes it when the product is inactive. If that fails, the answer is `503 search_index_failed`; the product is saved, and sending the same request again fixes the index. | ||
| +- `POST /admin/search/rebuild` (admin token) empties the index and fills it from every active product, 50 at a time, and answers `{ "indexed": n }`. Use it for a new search database, after `npm run db:seed`, or whenever results look wrong. While it runs, searches can miss products it has not reached yet. | ||
| + | ||
| +Image uploads and checkout do not touch the index: they change nothing search matches on. | ||
Changed.rebasely/kb/decisions/0001-d1-for-shop-data.md+2−1
| @@ -2,7 +2,8 @@ | ||
| id: 0001-d1-for-shop-data | ||
| type: decision | ||
| owner: justin | ||
| -status: accepted | ||
| +status: superseded | ||
| +superseded-by: 0011-search-index-in-its-own-database | ||
| reviewed: 2026-10-01 | ||
| --- | ||
| # 0001 One D1 database for the shop's data | ||
Added.rebasely/kb/decisions/0011-search-index-in-its-own-database.md+31−0
| @@ -0,0 +1,31 @@ | ||
| +--- | ||
| +id: 0011-search-index-in-its-own-database | ||
| +type: decision | ||
| +owner: justin | ||
| +status: accepted | ||
| +reviewed: 2026-10-16 | ||
| +--- | ||
| +# 0011 The search index gets its own D1 database | ||
| + | ||
| +## Context | ||
| + | ||
| +[[0001-d1-for-shop-data]] put everything in one D1 database, `storefront`, so checkout could take stock and create an order in one batch. Shoppers now need to search the catalog, and D1 has full text search through SQLite's FTS5, so the obvious place for the index is an FTS5 table next to `products`. | ||
| + | ||
| +Two things argue against that: | ||
| + | ||
| +- A D1 database runs one query at a time. Search is the busiest read the shop will have, since the search box asks on every pause in typing, and each request ranks matches with `bm25()`. In the same database, those queries wait in the same line as checkout's batches and the expiry cron's writes. | ||
| +- The index is derived data. It is rebuilt from `products` whenever the tokenizer changes, and nothing in it needs keeping. In the shop's database, changing it would mean a migration next to the ones that hold orders, and every rebuild would run its deletes and inserts on the database checkout writes to. | ||
| + | ||
| +## Decision | ||
| + | ||
| +The shop's data stays in one database, `storefront`, with everything [[0001-d1-for-shop-data]] said about migrations and batches. The search index lives in a second database, `storefront-search` (binding `SEARCH`), with its own migrations in `search/migrations/`. It holds only the text search matches on: a product's id, name and description, for active products. | ||
| + | ||
| +storefront-api writes the index right after every product write, and `POST /admin/search/rebuild` refills it from the catalog. A search finds ids in the index and reads the products from `storefront`. See [[search]]. | ||
| + | ||
| +## Consequences | ||
| + | ||
| +- Search load cannot slow down checkout, and the index can be dropped, retuned and rebuilt without touching the database that holds orders. | ||
| +- A product write is now two writes in two databases, with no transaction across them. If the index write fails, the merchant gets `503 search_index_failed` and sends the same request again; the upsert makes that safe. Anything that drifts anyway is fixed by a rebuild. | ||
| +- A stale index can only miss a product or return one that no longer matches. It can never show a wrong price or an inactive product, because results always come from the catalog. | ||
| +- Writes that bypass the API, such as `scripts/seed.sql`, do not reach the index. Run a rebuild after them. | ||
| +- Still one writer: only storefront-api binds either database ([[0006-one-writer-for-the-database]]). | ||
Changed.rebasely/kb/guides/local-dev.md+9−2
| @@ -6,7 +6,7 @@ | ||
| - wrangler.jsonc | ||
| - workers/*/wrangler.jsonc | ||
| - scripts/seed.sql | ||
| -reviewed: 2026-10-12 | ||
| +reviewed: 2026-10-16 | ||
| --- | ||
| # Running the shop locally | ||
| @@ -16,10 +16,17 @@ | ||
| ```bash | ||
| npm install | ||
| -npm run db:migrate # applies migrations/ to the local D1 | ||
| +npm run db:migrate # applies migrations/ and search/migrations/ to the local D1s | ||
| npm run db:seed # six products, one of them out of stock | ||
| echo 'ADMIN_TOKEN=dev-admin-token' > .dev.vars | ||
| npm run dev # storefront-api on localhost:8787 | ||
| +``` | ||
| + | ||
| +The seed writes straight to the database, so the search index is empty until you rebuild it: | ||
| + | ||
| +```bash | ||
| +curl -X POST localhost:8787/admin/search/rebuild -H 'authorization: Bearer dev-admin-token' | ||
| +curl 'localhost:8787/search?q=linen' | ||
| ``` | ||
| `curl localhost:8787/products` lists all six; the candle is out of stock, so adding it to a cart fails. Add `STRIPE_SECRET_KEY` (a test mode secret key) to `.dev.vars` to try checkout. | ||
Changed.rebasely/kb/guides/testing.md+3−2
| @@ -4,7 +4,7 @@ | ||
| owner: justin | ||
| describes: | ||
| - test/helpers/** | ||
| -reviewed: 2026-10-09 | ||
| +reviewed: 2026-10-16 | ||
| --- | ||
| # Running and writing tests | ||
| @@ -20,6 +20,7 @@ | ||
| `test/helpers/app.ts` builds the real fetch handler from `createApp` with a fresh environment per test: | ||
| - `DB` is `TestD1` (`test/helpers/d1.ts`), a small D1 stand-in over `node:sqlite` with every file in `migrations/` applied in order and foreign keys on. It covers `prepare`, `bind`, `first`, `all`, `run` and `batch`, and a batch is a transaction, as in D1. | ||
| +- `SEARCH` is a second `TestD1` with `search/migrations/` applied (`testSearchDb()`). Node's SQLite has FTS5, so the search tests run the real `MATCH` and `bm25()` queries. | ||
| - `IMAGES` is `TestR2` (`test/helpers/r2.ts`), an in-memory bucket whose `objects` map tests can inspect, and `IMAGES_URL` is `https://images.shop.test`. | ||
| - `ORDER_EVENTS` is `TestQueue` (`test/helpers/queue.ts`), which keeps every message in `sent`. | ||
| - Payments are `FakePayments` (`test/helpers/payments.ts`), which records every charge, fails the next one when `failNext` is set, and refuses to cancel the intents in `uncancellable`. `test/stripe-payments.test.ts` checks the real Stripe adapter against a fake `fetch`, so no test calls Stripe. | ||
| @@ -29,4 +30,4 @@ | ||
| ## Writing one | ||
| -Test behavior through the API rather than the modules, the way a client sees it. When code calls a D1 method the stand-in does not have yet, add it to `TestD1` in the same change. A new migration needs nothing: the stand-in applies whatever is in `migrations/`. See [[0001-d1-for-shop-data]] for why the SQL is shared. | ||
| +Test behavior through the API rather than the modules, the way a client sees it. When code calls a D1 method the stand-in does not have yet, add it to `TestD1` in the same change. A new migration needs nothing: the stand-ins apply whatever is in `migrations/` and `search/migrations/`. See [[0011-search-index-in-its-own-database]] for why there are two databases. | ||
Changed.rebasely/kb/index.md+3−1
| @@ -5,6 +5,7 @@ | ||
| ## Concepts | ||
| - [[catalog]]: products, prices and stock, and how the merchant edits them. | ||
| +- [[search]]: full text search over the catalog, and how the index stays in step. | ||
| - [[carts]]: what a cart holds and the limits it enforces. | ||
| - [[product-images]]: how images get into R2 and back out. | ||
| - [[checkout]]: how a cart becomes an order, and how stock and payment are handled. | ||
| @@ -14,7 +15,7 @@ | ||
| ## Decisions | ||
| -- [[0001-d1-for-shop-data]]: one D1 database for everything the shop sells. | ||
| +- [[0001-d1-for-shop-data]]: one D1 database for everything the shop sells (superseded). | ||
| - [[0002-serve-images-through-the-api]]: images were served by storefront-api (superseded). | ||
| - [[0003-money-in-whole-cents]]: prices and totals are integers. | ||
| - [[0004-take-stock-at-checkout]]: carts check stock, checkout takes it. | ||
| @@ -23,6 +24,7 @@ | ||
| - [[0007-at-least-once-with-idempotency-keys]]: repeats are expected and harmless. | ||
| - [[0008-images-from-the-bucket-domain]]: browsers load images from R2's custom domain. | ||
| - [[0009-cancel-unpaid-orders-after-30-minutes]]: unpaid orders give their stock back. | ||
| +- [[0011-search-index-in-its-own-database]]: the shop's data in one database, the search index in a second. | ||
| ## Guides | ||
ChangedREADME.md+5−1
| @@ -1,6 +1,6 @@ | ||
| # storefront | ||
| -The API behind a small online shop, running on Cloudflare Workers. Products, carts and orders live in D1, product images in R2, and payments go through Stripe. | ||
| +The API behind a small online shop, running on Cloudflare Workers. Products, carts and orders live in D1, the search index in a second D1 database, product images in R2, and payments go through Stripe. | ||
| There are three Workers: | ||
| @@ -27,6 +27,7 @@ | ||
| npm run db:seed | ||
| echo 'ADMIN_TOKEN=dev-admin-token' > .dev.vars | ||
| npm run dev | ||
| +curl -X POST localhost:8787/admin/search/rebuild -H 'authorization: Bearer dev-admin-token' | ||
| ``` | ||
| Product images are served from the bucket's custom domain, `IMAGES_URL` in `wrangler.jsonc`. Checkout needs Stripe: add `STRIPE_SECRET_KEY` to `.dev.vars` with a secret key from your Stripe account's test mode. | ||
| @@ -46,8 +47,10 @@ | ||
| | --- | --- | | ||
| | `GET /products` | Active products by name, a page at a time: `?limit=` (1 to 100, default 24) and `?after=` the previous page's `next` | | ||
| | `GET /products/:id` | One active product | | ||
| +| `GET /search` | Active products matching `?q=`, best match first, up to `?limit=` | | ||
| | `PUT /admin/products/:id` | Create or replace a product (admin token) | | ||
| | `PUT /admin/products/:id/image` | Upload the product's image (admin token) | | ||
| +| `POST /admin/search/rebuild` | Refill the search index from the catalog (admin token) | | ||
| | `POST /carts` | A new, empty cart | | ||
| | `GET /carts/:id` | A cart with its lines and totals | | ||
| | `PUT /carts/:id/items/:product` | Set a quantity, `{ "quantity": n }` | | ||
| @@ -63,6 +66,7 @@ | ||
| src/ storefront-api: router, catalog, carts, images, checkout, orders | ||
| workers/ the other Workers, each with its own wrangler.jsonc | ||
| migrations/ D1 schema, applied in order | ||
| +search/ the search index's schema (search/migrations/) | ||
| test/ node --test suites and the D1 and R2 stand-ins | ||
| .rebasely/ architecture map, knowledge base and CI workflow | ||
| ``` | ||
Checks
1 of 1 check failed| Status | Check | Reported by | Description | Details |
|---|---|---|---|---|
| Failed | duragit/ci/test | Rebasely Actions | out of memory | Run |
Runs
| Status | Workflow | Event | Started |
|---|---|---|---|
| Failed | ci | Push |