Suppose we want to get the number of replies to a post.

We could do COUNT (same as REQ but returns a counter) as is proposed by NIP-45.

Problems:

  • you can't query several counts at once, the way clients need it to fill counters for the scrolled feed
  • hard-ish to implement optimally - relays need to become smarter
  • smart relays/aggregators become single points of failure (SPOFs)

What if counters were themselves nostr events, published by some data app? It could have it's own special-purpose relay and could generate these events on the fly when requested with REQ, or it could be a background process that does the calculations and then just publishes results on it's own off-the-shelf relay.

Benefits:

  • relays stay dumb
  • many counters can be requested with one REQ
  • counters can be cached/forwarded between relays and the data app is less of a SPOF
  • yet cached/forwarded counters can't be tampered, they're signed and timestamped

Cool!

Here is the next issue. I want to get top profiles by some 'popularity' metric for user discovery.

Options:

  • download all user profiles from a relay and sort on the client - too expensive
  • rely on a third-party non-nostr api - not cool
  • add some modifier to REQ to override the default sorting by created_at - also bad (tried it):
    • can't do pagination, because the only pagination that can be done with REQ is using until/since
    • relay has to be very smart - SPOF

What if the promising idea with generating counters as nostr events could be extended to support any computation, not just counting?

Generic Nostr Data App Framework

Let's solve for 'counters' first, where the result of a computation is small (does not require pagination).

There are NIP-33 parameterized replaceable events, where the d tag is used as an identifier, so that a newer event with the same kind, pubkey and d tag replaces the older event. It seems to fit nicely: we could pass a list of parameters as a d tag, and each invocation of our computation with the same parameters would produce a new event that replaces the previous one.

Let's use kind:33333. Here is how a client that wants a counter of likes of event 'E' from a data app 'A' could request it from the data app's relay:

["REQ", "", {"authors":[A],"kinds":[33333],"#d":["m=count&k=7&e=E"]}]

Notice that d tag contains parameters: m=count (method name), k=7 (kind=reactions) and e=E (our target event 'E').

And the result for such a request, whether generated by A's smart relay on the fly, or pre-calculated and published on A's dumb relay, is:

{"id":I,"pubkey":A,"kind":33333,"created_at":T,"tags":[["d","m=count&k=7&e=E"]],"content":"{\"count\":100}","sig":"..."}

What if the counter changes? Our data app A would publish a new event with an updated counter, and all clients that are still subscribed to the same kind/author/#d filter will receive it!

What if client wants to fetch several counters at once? It would just specify several #d tag filters, and would fetch those counters as they're published by the data app.

What if people don't want to depend so heavily on the data app's relay? They can fetch some counters and forward them to their own relay, and then use it as a fallback if the data app is down. The events are signed and timestamped, so clients can verify who produced some computation, and how stale it is.

Sounds good so far.

Pagination

Suppose I want the list of top profiles, kind:0 events ordered by some popularity metric. The results are potentially huge, we can't put it all into a single result event, so we need pagination.

To keep the same logic of parameterized replaceable events, the simplest thing seems to be to just add an extra parameter that specifies the page number, i.e. &page=0 for the first page, &page=1 for the second etc. This way the new pages would replace the older ones, and clients could query several pages with one #d filter.

Here is how our request for top profiles could look like:

["REQ", "", {"authors":[A],"kinds":[33333],"#d":["k=0&sort=popular&page=0"]}]

Here is the first page of results, an event with 3 links:

{"id":I,"pubkey":A,"kind":33333,"tags":[["d","k=0&sort=popular&page=0"]],"created_at":T,"content":"\[E1,E2,E3\]"}

Note that the list of event ids is encoded in the content field, not in tags. This is to make sure that
these events are only query-able using the d tag, and to avoid forcing dumb relays to index these lists. Plus,
this is a generic framework, so results can be anything, not just links to events or pubkeys.

To signify the end of pages, the data app should use some flag in the content of last page.

If the list changes, the data app A would publish a new set of events (pages) with same set of d tags to
replace the previous results.

One issue here is that if old results had more pages than the new results, some old pages would be left un-replaced. I haven't figured out a proper solution here, so your suggestions are welcome.

Parameter Normalization

One issue with using d tag to pass parameters and replace old results is that the same set of parameters can be encoded in different ways and produce different d tags. I suggest applying some normalization: convert all params into percent-encoded form, then sort params as strings, and only then concatenate them into a query string. This would ensure that the same set of params produces the same d tag.

The First Nostr Data App (Ndapp)

A working implementation of this concept can be found at https://data.nostr.band. Please also comment on the proposed NIP for this.

Nostr.Band is an aggregator, so if this approach is supported by the community, and if other aggregators decide to build their own ndapps, we could then standardize some set of aggregation layer methods. This way clients could switch between aggregators, without having to rely on non-nostr custom APIs.

Nostr Data Apps