Reframing WIT as primarily a machine format
— 2023-11-19

This is a "short" post. Unlike my usual (lengthy) research pieces, this is mostly a train-of-thought, low-edit post intended to convey an idea or perspective I thought would be useful to write down.

edit(2023-11-19): I've changed the title of this post from "Rethinking WIT" to "Reframing WIT". I strongly hold that WIT being human-readable is a good thing we shouldn't change, even though in this post I make the case that despite that it probably shouldn't be the primary way we interact with it.

Earlier this year I wrote about how I believe HTML is too complex to be feasibly written by hand, and rather than linting manually authored HTML for errors it's probably more effective to treat HTML as a target which should be compiled to. I ended up putting these ideas to practice in the html crate for Rust, and I think that ended up being roughly the right idea. At the core of this was a reframing of "HTML" as this thing that's meant to be read and written by humans, to a format that is primarily intended to be produced and ingested by machines.

Recently I've been using WIT - the IDL format used to define Wasm Components with - and I'm slowly starting to form opinions about it. Yesterday I wrote about why I believe ABIs can be described as: "A type system + object encoding + calling convention". But today I want to take a slightly different perspective on WIT and share an opinion I've started forming: I believe that WIT should primarily be produced and consumed by tools, and not by humans. This is a departure from how people have been reasoning about WIT so far - which is to think of WIT as a human-readable format people will want to read and write by hand. In this post I want to explain why I believe this is probably a better way to reason about WIT.

Interacting with WIT

I believe that from a programmer's perspective all WIT usage can be roughly broken up into two categories: consuming WIT definitions to use in programs. And producing WIT definitions to be used in other programs, or to run your program in a certain runtime. I believe that when we import WIT, we probably don't want to read WIT files directly. What we're more likely interested in are the bindings which would be generated for our programming language. Or perhaps if we're familiar with WIT itself, we might be interested in an interactive doc explorer a la rustdoc to navigate the type hierarchy.

If we're going to be exporting code using WIT, then we probably want to start by writing the code itself. In Rust when we want to serialize types we don't typically start writing a JSON Schema - instead we start by writing our structs and adding a serde derive at the top. And we don't just do this for JSON, we take a similar data-first approach for command lines too. I believe Wasm Components and their WIT definitions would likely do well with a similar approach. Because integrating it into the language like this is by far the path of least resistance.

I believe this will probably hold true for the majority of programmers. Rather than writing or consuming types authored in WIT, it'll be much easier to reason about components in terms of how they integrate into the language. And if for some reason reasoning about the bindings is not sufficient, a documentation tool will probably be the easier to way to explore WIT APIs directly.

Why perspectives may differ

WASI Preview 2 hasn't been released yet, and so up until this point the majority of people working with WIT haven't actually been programmers trying to build things with WIT. So far the people using WIT have mostly been the people working on WASI itself, primarily working to define the language-agnostic interfaces everyone will be writing their programs against.

And so I think this is a likely explainer for why the readability of WIT has been highly valued so far. The work the majority of people using WIT today is to write WIT by hand. But I believe that once Preview 2 releases, this will likely become a minority use case. If done right, most people working with WIT will be using it via programming languages instead.

Even for system rewrites I believe it's likely easiest to generate bindings for an existing impl directly from a language. Then take the WIT definitions generated by that, and use that as the interface for the new implementation. At no point does this require people to directly interact with WIT definitions.

On tradeoffs

To me it makes sense that if we believe that the primary way people use WIT is either write it by hand, or read the source files, that the property of "readability" is one we're least keen to compromise on. I'm not quite sure about the exact tradeoffs; but my understanding is that the "readability" of WIT has been fairly important.

So far I've been making the case that I believe people are more likely to be ingesting WIT using language-specific bindings, producing it via language ascriptions, and reading it via documentation tools. And so if that's the case, then "readability" may perhaps be something we're more willing to compromise on in favor of other properties.

That's not that I'm trying to argue that being able to write WIT by hand or read it directly isn't important. What I'm trying to say is that I don't believe it necessarily is the most important property. There may be other properties, like expressivity or machine-parsability, which would compromise readability. If we value "readability" differently, then we may very well end up making different decisions.

Where to go from here?

I believe that the more closely we can integrate Wasm Components/WASI with language toolchains and guest languages, the easier it will be for people to adopt it. The explainer I've used for this in the past is in terms of "deltas". The less people need to deviate from their existing workflows to do a new thing, the more likely it is they'll want to try the new thing. Applied to Components: the more it feels like importing and exporting Components is like using any other types in that language, the more likely it is that people are willing to use it.

With that in mind, I would like us to start thinking about ways in which we can remove manually authored WIT entirely from the end-to-end experience. If done right, both importing and exporting WIT definitions from guest languages should feel completely native. As if we were importing or exporting any other library written in that same language. I feel like we're close to that for some languages ¹, but I believe we can probably improve on it.

I want to add here: the existing efforts I've seen on this front are really impressive! I'm in no way trying to snub the existing work done!

The other thing I think we should explore is a native wit-doc tool which can be used to render WIT definitions to HTML. My thinking is that this should be modeled loosely after rustdoc or Swift's DocC. Given plans are plans to make progress on the Warg registry next year - this seems like something which could integrate nicely with that too. Over in Rust land, one of my least favorite splits is between "docs.rs" and "cargo.io" - in my opinion it would make sense to merge these two. And given wit-bindgen already runs in the browser, we could probably also generate previews of what generated output bindings for various programming languages would look like. Not unlike the SDK example generators used in some of the world's fanciest docs.

Conclusion

In this (short) post I've tried to make the case that while historically WIT has been authored by hand and read from source files, that may not necessarily be continue being true in the future. To me it seems more likely that once Preview 2 is released, people using Wasm Components will primarily be interested in using Wasm Components directly from source languages. Since that is the path of least friction. And in the case people need to look up WIT APIs directly, a (hosted) documentation tool will probably provide a better experience than reading files directly from source.

I argue that if we no longer believe that people will primarily interact with WIT by manually authoring WIT files, and reading them from source, that the "readability" property of the WIT format may become less important. That's not to say it should ever be deemed unimportant, since there are valid reasons to interact directly with WIT files. But it may be the case that if change how we value this property, we will become more willing to trade it off in favor of other properties.