Extend the Web Forward

Yehuda Katz -

If we want to move the web forward, we must increase our ability as web developers to extend it with new features.

For years, we've grabbed the browsers extension points with two hands, not waiting for the browser vendors to gift us with new features. We built selector engines, a better DOM API, cross-domain requests, cross-frame APIs.

When the browser has good extension points (or any extension points, really), we live in a virtuous cycle:

  • Web developers build new APIs ourselves, based on use-cases we have
  • We compete with each other, refining our libraries to meet use cases we didn't think of
  • The process of competition makes the libraries converge towards each other, focusing the competition on sharp use-case distinctions
  • Common primitives emerge, which browser vendors can implement. This improves performance and shrinks the amount of library code necessary.
  • Rinse, repeat.

We've seen this time and time again. When it works, it brings us querySelectorAll, the template element, and Object.observe.

The Sad Truth

The sad truth is that while some areas of the browser are extremely extensible, other areas are nearly impossible to extend.

Some examples include the behavior and lifecycle of custom element in HTML, the CSS syntax, and the way that the browser loads an HTML document in the first place. This makes it hard to extend HTML, CSS, or build libraries that support interesting offline capabilities.

And even in some places that support extensibility, library developers have to completely rewrite systems that already exist. For example, John Resig had to rewrite the selector engine from scratch just to add a few additional pseudo-properties, and there is still no way add custom pseudo-properties to querySelectorAll.

Declarative vs. Imperative

A lot of people see this as a desire to write everything using low-level JavaScript APIs, forever.

No.

If things are working well, JavaScript library authors write new declarative APIs that the browser can roll in. Nobody wants to write everything using low-level calls to canvas, but we're happy that canvas lets us express low-level things that we can evolve and refine.

The alternative, that web authors are stuck with only the declarative APIs that standards bodies have invented, is too limiting, and breaks the virtuous cycle that allows web developers to invent and iterate on new high-level features for the browser.

In short, we want to extend the web forward with new high-level APIs, but that means we need extension points we can use.

Explaining the Magic

If we want to let web authors extend the web forward, the best way to do that is to explain existing and new high-level forms in terms of low-level APIs.

A good example of in-progress work along these lines in Web Components, which explains how elements work in terms of APIs that are exposed to JavaScript. This means that if a new custom element becomes popular, it's a short hop to implementing it natively, because the JavaScript implementation is not a parallel universe; it's implemented in terms of the same concepts as native elements.

That doesn't necessarily mean that browsers will simply rubber-stamp popular components, but by giving library authors the the tools to make components with native-like interfaces, it will be easy for vendors to synthesize web developer zeitgeist into something standard.

Another example is offline support. Right now, we have the much-derided AppCache, which is a declarative-only API that makes it possible to display an HTML page, along with its assets, even if the browser is offline.

AppCache is not implemented in terms of a low-level JavaScript API, so when web developers discovered problems with it, we had no way to extend or modify it to meet our needs. This also meant that we had no way to show the browser vendors what kinds of solutions would work for us.

Instead, we ended up with years of stagnation, philosophical disagreements and deadlock between web developers and specification editors, and no way to move forward.

What we need instead is something like Alex Russell's proposal that allows applications to install JavaScript code in the cache that intercepts HTTP requests from the page and can fulfill them, even when the app is offline. With an API like this, the current AppCache could be written as a library!

Something like Jonas Sicking's app cache manifest is a great companion proposal, giving us a nice starting point for a high-level API. But this time if the high-level API doesn't work, we can fix it by using the low-level API to tweak and improve the manifest.

We can extend the web forward.

Extensions != Rewriting

It's important to note that web developers don't want a high level API and then a cliff into the low-level API.

Today, while you can implement custom elements or extend the selector engine, you can only do this by rewriting large chunks of the stack alongside the feature you want.

Real extensibility means an architecture that lets you tweak, not rewrite. For example, it would be possible to add custom rules to CSS by writing a full selector engine and application engine, and apply rules via .style as the DOM changes. With mutation observers, this might even be feasible. In fact, this is how some of the most devious hacks in the platform today (like the Polymer Shadow DOM polyfill) actually work.

That kind of "extensibility" doesn't fit the bill. It doesn't compose well with other extensions, defeats the browser's ability to do performance work on unrelated parts of the stack (because the entire stack had to be rewritten), and is too hard to provide meaningful iteration.

Browser implementers are often wary of providing extension points that can be performance footguns. The biggest footgun is using libraries that rewrite the entire stack in JavaScript, and whole-stack-rewriting strategies are the tactic du jour today. For performance, we have little to lose and much to gain by making extensions more granular.

Extend the Web Forward

So what do we gain from a more extensible web? I'll let Dave Herman, a member of TC39, answer that for me.

  • When you design new APIs, you are forced to think about how the existing system can express most of the semantics. This cleanly separates what new power is genuinely needed and what isn't. This prevents cluttering the semantics with unnecessary new magic
  • Avoiding new magic avoids new security surface area
  • Avoiding new magic avoids new complexity (and therefore bugs) in implementation
  • Avoiding new magic makes more of the new APIs polyfillable
  • Being more polyfillable means people can ramp up faster, leading to faster adoption and evolution of the platform
  • Avoiding new magic means that optimizations in the engines can focus on the stable core, which affects more of new APIs as they are added. This leads to better performance with less implementation effort
  • Avoiding new magic means less developer education required; people can understand new APIs more easily when they come out, because they build off of known concepts
  • This means that the underlying platform gets fleshed out to be expressive enough to prototype new ideas. Library authors can experiment with new features and create more cowpaths to fill the Web API pipeline

All this, and more! There's something for everybody!

Implementors and web developers: let's work together to extend the web forward!