Introducing Pipette

Hello from my temporary location in East Vancouver!

My name is Jim Pick. I’m a freelance software developer that lives in Vancouver, Canada.

“Pipette” is the name of the new open source project I’ve been working on for the past few months.

And this blog is where I’m going to maintain a development journal.

So what is Pipette?

Pipette is a blogging system.

So how is this different than other blogging systems that exist today?

Pipette is built using “Peer-to-Peer Web” technologies.

The blogs that you create with Pipette are published as Dat Archives, which can be synced to any computer and even viewed when you are offline.

For example, this blog is published as a Dat archive with the following address:

dat://cf1c6948c28017a94f7ffbbcd836b9cc3b6e15f328cb1eec5d91fd5f7109f969/

Don’t worry … you can make less scary names for sharing. And you can also publish on a traditional web address.

You can create a new blog directly from within Beaker Browser and publish immediately, straight from your browser. Creating a new blog is practically free. And you don’t have to upload your content to a centralized hosting service like Facebook, Medium or Tumblr.

Note: At this point in time, Pipette is only at the proof-of-concept stage, so there are a couple of manual one-time setup steps that are not yet automated.

Is it anything else?

So glad you asked! Pipette is also an experiment in cloneable/forkable service infrastructure.

We can make it possible for individuals and communities to have the ability to easily fork an entire web service. Not just the source code or the final HTML/CSS/JavaScript of a web service … but also all of the intermediate machinery running in the cloud needed to support the web service.

If we enable ‘forking’ on a deeper level, we open up a whole new way for people to work together to collaboratively run and maintain complicated services.

It’s a deep topic, so I’ll elaborate more in a future post.

Why the name “Pipette”?

Since much of the code runs in Beaker Browser, and a “Pipette” is also a piece of laboratory equipment … it seemed to fit.

A Pipette (image from Wikipedia)

Is this blog created using “Pipette”?

Yes! This is first blog created with it. I’ll eventually move my personal blog over to use it as well.

Is there a demo?

I’m going to put a full demo together in the next few days.

I can’t wait. Can you give me a quick breakdown of how to make a new blog?

Sure.

First, you need to install Beaker Browser.

Second, you need to use Beaker to “fork” a copy of the “CMS” (content management system) which you will use to write the content of your blog posts. If you are familiar with WordPress, the CMS is basically the dashboard – except in this case, it lives at a separate website.

I’m using a modified version of Netlify CMS which knows how to write back to it’s own Dat archive (self-modifying pattern) instead of writing to Git (a version control system for programmers).

Netlify CMS Screenshot

You can use Beaker to fork the CMS for this blog as a starting point:

dat://d6185c1680001cd2260a0f31bfb209edbf97551774dc6175c110019d9020199d/

After you fork it, you will probably want to delete the existing content and change the blog title in the Settings menu.

Third, once you’re written your first blog post, you’ll want to turn it into a full static web site suitable for publishing. In order to do that, the initial version of Pipette is using the Hugo static site generator. It’s nice and fast. It’s also popular and has most of the features you’d want in a blog, such as an RSS feed.

I’ve written a “worker” program that can easily be installed onto a Linux virtual machine that will subscribe to changes on the source CMS Dat archive, and generate or update the static website in a separate output Dat archive whenever a blog post is created or edited. Here’s a (slightly scary) look at the worker as it does its thing:

Hugo worker process rebuilding a blog after a revision

Notice that it’s quite fast.

The nifty thing about the “worker” is that it is packaged up as a full Linux virtual machine image (using @mafintosh’s dat-container) - and is itself distributed via Dat. The whole thing fits inside a 31MB file: Linux, Hugo, Node.js and all the custom logic!

Fourth, the Peer-to-Peer web works much like BitTorrent – there’s no single server, just a bunch of individuals sharing copies. You probably want to get some additional “peers” for your new blog so that multiple copies of it are available in case some go offline.

And you probably also want to expose it to regular web browsers via HTTP/HTTPS. Here is where publishing to Dat shines!

You can easily set up another peer and get a free website all at once using Hashbase from the creators of Beaker Browser. This blog is published using that option: https://pipette-dev-blog-jimpick.hashbase.io/

Alternatively, you can install a dathttpd server on your own hardware.

Another approach would be to write some software that subscribes to the Dat archive with the rendered static files, and republish those to an existing CDN or static file hosting service, such as GitHub Pages, Zeit Now, Surge, Netlify, Aerobatic, Firebase Hosting, or even raw Amazon S3 or a CDN such as Amazon CloudFront. There are a huge number of options.

Is there source code?

Yes! But not much documentation yet.

I’ve actually been working on parts of this project for the past few months. There are a large number of parts to document, so it will take a while. I’m going to use this blog to write about the bits involved over the next few weeks.

The main project currently lives at:

I’ll probably rename it once I document it some more. I haven’t even updated the README for it. There are several Linux dependencies to install first, and then it can be installed using npm -g dat-hugo-worker. It needs to be run as root since it starts a new Linux virtual machine using the systemd-nspawn virtualization system. It is invoked as dat-hugo-worker dat://.... - which will cause it to subscribe to the Dat file provided. Assuming that it’s one of the Netlify CMS based Dat files with some blog posts, it will create a new Dat archive with the output and share it.

This is the actual worker that runs inside the dat-container based virtual machine image.

This is the modified version of Netlify CMS.

A modified version of this wrapper for Hugo (from Netlify) also lives inside the virtual machine image.

Next Steps

The big thing now is documentation!

The development and build process is many-layered. There are multiple images, and they are currently hand-built. There needs to be some more automation and testing. If I don’t document the build process, even I might forget all of the manual steps involved.

Most people aren’t going to want to set up a Linux box. Really, most bloggers won’t even know what Linux is! The worker only subscribes to one source Dat at a time, but it would be really nice if it could subscribe to many more than that. It would be easy to build a web service that could provide Hugo workers for hundreds or thousands of blogs from a single Linux instance.

I’d really like to build an Electron app that would allow people to run the worker on their own computers at home or at work.

Supporting Pipette

I’m releasing everything as open source.

If you can help test or help with documentation or adding features, contributions are encouraged!

Given my background working for large storage and cloud hosting companies in the past, I’ve been thinking about what a business for hosting dat-containers might look like. If you are interested in talking to me about that, I’d love to chat!

I support my family with freelancing work. If you are looking for somebody to do some Node.js development, DevOps, or even Dat ecosystem work, I’m your guy! Right now, I have availability. Try pinging me on Twitter at @jimpick, or via email at jim@jimpick.com.