More funding, new milestones

April 12, 2023

Good news: NLnet Foundation decided to extend their small grant for LibResilient! That grant funded bulk of the work over the last year and a half. It will now continue to do so for the next year or so.

The grant extension also helped to define the milestones for 2023. Here they are, in no particular order — the order they get implemented depends on many factors, including some non-obvious interplay between them. Some of these milestones are pretty simple, some will require substantial re-writes. Exciting times ahead!

Improved user experience

LibResilient needs a “still loading” screen to be displayed when loading HTML resources over slow transports. For example, retrieving content from IPFS can take upwards of 20s to load sometimes. Currently the user experience here is lacking: depending on browser timeout defaults, the page fails to load, showing an obscure error, and then suddenly loads and displays content.

Also, LibResilient currently kicks-in on the second request to a site that uses it, as that’s when the ServiceWorker gets actually loaded. This could be improved by using Clients.claim().

Plan

• implement a “still loading” screen; research questions:
  • will the return temporary HTML content need to explicitly redirect?
  • or will the browser redirect by itself?
  • which HTTP codes to use?
• test Clients.claim() use and implement if makes sense

MIME-type deduction

When retrieving content using certain transports (for example IPFS), MIME-type information is not available. Currently, plugins that face this issue naïvely try to guess the MIME-type based on file extension. What is needed is a facility to reliably establish MIME-types of requested content, made available by the ServiceWorker itself to all plugins that need it.

Plan

• implement MIME-type deduction directly in the ServiceWorker code
  • research if it would be possible to achieve this by looking at actual content, using some MIME-sniffing JS library
• define an internal API for plugins to use when they need it
• update plugins which currently use the naïve approach to use the new internal API
• document all of this properly

CORB/CORS/CSP-related testing and documentation

The interplay between LibResilient, CORB, CORS, and CSP is not well documented.

Documenting it well requires substantial testing on small but purpose-built infrastructure.

Plan

• design and deploy infrastructure for testing CORB/CORS/CSP in the context of LibResilient
• test different settings and edge-cases
• document best practices and potential pitfalls

Cookies/credentials-related testing and documentation

The assumption is that cookies and other credentials should not be exposed to alternative transports (for example), but this needs strict testing and documentation. Perhaps this should also be configurable.

Plan

• design and deploy infrastructure for testing this
• test settings, assumptions, and edge-cases
• document best practices and potential pitfalls
• potentially (if deemed useful) implement configuration options for sending cookies/credentials when retrieving content via alternative transports

Rewriting tests for Deno

Currently LibResilient uses Jest and Node.js for tests of the browser-side code, and Deno for the CLI and CLI-related tests. This makes maintenance difficult. Deno is a much better choice as it implements WebAPIs natively. So, browser-side code tests should be re-written for Deno, and Node.js dependency completely removed from te project.

Plan

• rewrite browser-side code tests for Deno
• make sure all integrations with CI/CD set-up work

Improved error handling

LibResilient has to handle request errors better. It needs to be smart about displaying the original 404 page from the original domain, and otherwise needs to show some form of a plugin call stack or other explanation if a request failed. Perhaps a “development” mode should be implemented. Currently when a request fails (for example, due to 404 error, or because integrity check fails, or…), a browser-internal “request failed” page is displayed to the user. This is not very helpful when debugging issues, and at the same time this is confusing to users.

From the developer (and user) perspective, it’s difficult to figure out what went wrong when a resource is not successfully fetched, and if the failure is related to LibResilient or not. This is especially problematic when fetching resources that are protected by subresource integrity.

This will require substantial rewrites of crucial pieces of the ServiceWorker and plugins.

Plan

• design the error handling system for LibResilient
  • plugins throw standardized JS errors or otherwise return standardized error responses
  • ServiceWorker handles them and potentially translates them to HTML error pages to display to the user
• implement it in ServiceWorker
• implement it in plugins
• consider implementing “developer mode”
• document the error handling system for LibResilient
• document: best practices regarding when what kind of errors should be thrown by plugins
• document: how the ServiceWorker handles those errors

Improved config.json handling when fetched via alternative channels

When loading config.json via alternative transports, if config.json is broken but the original website is not available, a LibResilient-enabled site might end up in a broken state

LibResilient should verify that a newly loaded config is valid broken (say, by deploying the new configuration and attempting to load the config.json file it just loaded), and reverting to the previous, clearly working config otherwise.

Plan

• test possible failure modes of config.json loaded via alternative transports
• improve handling of broken/problematic config.json
  • consider implementing deep plugin check
• update documentation

Improving experience for site administrators

There are several “papercut” issues in LibResilient and its plugins that are too small to be considered a separate project plan items, but are nonetheless important to fix.

Plan

• basic-integrity plugin: how to treat URIs without domain name?
• ServiceWorker: deterministically sorting query parameters
• create and test code to remove the service worker

DNSlink-based plugins: DNS-over-HTTPS without JSON

DNSlink-based plugins as currently implemented can only use DNS-over-HTTPS servers that offer JSON endpoints. Implementing pure DNS-over-HTTPS would greatly (several orders of magnitude) improve the number of DoH servers DNSlink-based plugins can use.

Plan

• implement pure DoH in DNSlink-based plugins
  • dnslink-ipfs
  • dnslink-fetch
• update documentation