Literate programming in Go | InfoWorld

Throughout a latest firm hackathon, my group’s constitution was to enhance the documentation for the Steampipe plugin SDK. Like different elements of the Steampipe system, the plugin SDK is written in Go and revealed to pkg.go.dev. The model that existed once we began is here. As is typical, the documentation was an autogenerated catalog of capabilities and kinds. To clarify the way to use these capabilities and kinds, we supplied guidance on the steampipe.io website.

Our hackathon problem was to weave such steerage into the generated documentation. Should you’re writing a Steampipe plugin, your checklist of duties appears like this:

1. Outline the plugin
2. Create the plugin entry level
3. Outline your first desk
4. …

These duties require that you just use capabilities and kinds, however whereas feedback connected to these capabilities and kinds can improve the generated documentation, they’re too granular for the high-level exposition we aimed for. Looking for inspiration, Steampipe lead developer Kai Daguerre seen that our top-level web page lacked the overview part he noticed within the documentation for pgx, a Go driver for Postgres.

That overview comes from https://github.com/jackc/pgx/blob/master/doc.go, which is one lengthy remark (that makes use of Go comment syntax) adopted by a package deal declaration.

So we added a doc.go on the prime degree of our repo to provide this overview.

IDG

Writing a wiki in Go feedback

We used the title doc.go as a result of that appears to be typical, however it might have been referred to as foo.go. What’s salient is that it is a legitimate Go file that belongs to a package deal. The package deal comprises no code, solely documentation. We wrote headers to create the sections of the overview, and in every part we described a plugin author’s activity utilizing narrative, inline code examples, inner hyperlinks to capabilities and kinds, and exterior hyperlinks to examples elsewhere.

The flexibility to hyperlink throughout the code’s namespace was a revelation. To explain the duty referred to as Add hydrate capabilities, for instance, we wrote this:

# Add hydrate capabilities

A column could also be populated by a Checklist or Get name. If a column requires information not present by Checklist 
or Get, it might outline a [plugin.HydrateFunc] that makes an extra API name for every row.
Add a hydrate operate for a column by setting [plugin.Column.Hydrate].

The part outlined by the Add hydrate capabilities header is a hyperlink goal: Add hydrate functions. And the bracketed objects render as hyperlinks to a sort, plugin.HydrateFunc, and to a property, plugin.Column.Hydrate. This was beginning to really feel like wiki writing!

What we nonetheless lacked, although, was the power to create new wiki pages the place we might clarify higher-level ideas. The overview was one place to do this, however we needed to order that for narration of the plugin author’s journey. The place might we talk about an idea like dynamic tables, a complicated characteristic that permits plugins like CSV which haven’t any fastened schema and should outline columns on the fly?

Kai realized that not solely might we create new documentation-only packages for such matters, we might additionally import them in order that their names have been obtainable for a similar form of shorthand linking we might do with, e.g., [plugin.HydrateFunc]. Within the top-level doc.go he did this:

package deal steampipe_plugin_sdk

import (
	"github.com/turbot/steampipe-plugin-sdk/v5/docs/dynamic_tables"
)

var forceImportDynamicPlugin dynamic_tables.ForceImport

And in /docs/dynamic_tables/doc.go he did this:

package deal dynamic_tables

sort ForceImport string

Now we might write a sentence in a remark like this:

/*
Use [dynamic_tables] once you can't know a desk's schema prematurely.
*/
package deal steampipe-plugin-sdk

And the bracketed time period autolinks similar to every other title within the code’s namespace.

If that looks as if extra hassle than it is value, you possibly can skip the import gymnastics and simply use an exterior hyperlink:

/*
Use [dynamic_tables] once you can't know a desk's schema prematurely.


[dynamic_tables]: https://pkg.go.dev/github.com/turbot/steampipe-plugin-sdk/v5/docs/dynamic_tables
*/
package deal steampipe-plugin-sdk

Both manner, the authoring expertise now feels extra absolutely wiki-like. You may create new pages as wanted, and weave them into the hypertextual documentation that lives throughout the code base.

It was, admittedly, a little bit of a battle to make use of the Go system within the methods described right here. The remark syntax is Markdown-like however frustratingly not Markdown; it depends upon many implicit formatting conventions. Should you go this route you will want an area previewing device, and it is not super-obvious that the one you need just isn’t godoc however slightly pkgsite. Had we found the command go set up golang.org/x/pkgsite/cmd/pkgsite@newest we’d have saved ourselves quite a lot of grief.

How is that this literate programming?

It is not! In his eponymous paper Knuth wrote:

The practitioner of literate programming may be considered an essayist, whose predominant concern is with exposition and excellence of fashion. Such an writer, with thesaurus in hand, chooses the names of variables fastidiously and explains what every variable means. She or he strives for a program that’s understandable as a result of its ideas have been launched in an order that’s finest for human understanding, utilizing a combination of formal and casual strategies that reinforce one another.

To help this apply he invented a system referred to as net whose elements, tangle and weave, enabled the writer of a program to inform its story in a language that blended code (initially, Pascal) and documentation (initially, TeX) in a narrative-first manner. Our trendy methods of blending code and documentation, and producing docs from embedded feedback, solely superficially resemble Knuth’s apply, as Mark Jason Dominus identified in his essay POD is not Literate Programming (2000).

And but, now that our hackathon train has given me a style of what code-as-wiki may be, it does really feel like a step towards the storytelling method that Knuth advocates. Recall that he named his system net earlier than there was the online we now inhabit, by no means thoughts wikis! I can not declare that we’re now literate programmers within the Knuthian sense, and I’ve at all times puzzled if solely a Knuthian thoughts might implement that unique imaginative and prescient. However this fashion of bettering the narrative high quality of autogenerated docs does really feel like a step in the precise course.