TiddlyWiki

Mar 3, 2014 at 2:35 PM
Edited Mar 3, 2014 at 2:36 PM
It was trivial to CouchDB-enable (and thus Thali-enable) LittleOutliner. TiddlyWiki is next up, and it's a more challenging port. I hadn't looked at it in a long time. The version now in beta (TW5) is, Jeremy Ruston says, "a major reboot for the next 25 years." It's a /very/ sophisticated single-page app that can also run under node.js. In that mode it syncs changes to the webserver's filesystem. I'd like to redirect it to CouchDB (and then Thali).

I hadn't debugged node.js apps before, and this is a complex one. I l tried Chrome, node-inspector, and WebStorm, but am having the best luck with https://nodejstools.codeplex.com/.

If this works, we'll have a version of TiddlyWiki that anybody could use with CouchDB to gain the benefits of replication and versioning. And we'll have a version that uses Thali for additional identity and security benefits. And those two things should be "the same" in some important sense that I can't define yet.

As an aside, the TW5 architecture is quite instructive. For example, it can load, execute, and export JavaScript modules from the DOM or the filesystem, depending on the mode (browser vs node). And those modules are themselves tiddlers viewable within the system as wiki pages. Very cool.
Mar 3, 2014 at 4:31 PM
I should warn you that we don't generally have node support in Thali land. There are several reasons for this:
  1. Node is a server with a ton of functionality and dependencies and I'm not sure it makes sense for us to add add a dependency that large.
  2. Node's support on Android appears to be questionable. The only project I've seen is anode which was last updated six months ago. It is based on a port of the core node code (see here) that was last updated 4 months ago. Compare this change log from anode with this change log from Node.Js and you get a sense for how old the code is.
I actually think providing Node.JS on Android for projects like Thali makes a ton of sense. I'd love to see it happen. But you know we already have more work than we can handle as is so we simply can't take this on. Something that big would need to be a well resources, well maintained project.

So what's that story without node?

BTW, as a side note, PouchDB ran in server mode using Node.Js as well. When I dug into it I found that it's node dependencies were really shallow (basically receiving incoming connections and some URL routing) so I was able to rewrite those dependencies in general Javascript and use the Javascript/Java bridge to handle the heavy lifting. The whole thing took less than 2 weeks. So just because a code base uses node it doesn't necessarily mean show over. We just have to evaluate how much work it is to port the parts of node they use to our environment.
    Thanks,

         Yaron
Mar 3, 2014 at 5:07 PM
Edited Mar 3, 2014 at 5:49 PM
Understood. TiddlyWiki by definition doesn't need Node.js because it can work as a pure single-page browser app. In theory I "just" need to inject an adapter. TW has a Node-based build script that packages up all the JS code for various uses, some being server modes and others serverless. I don't think much code need be written for a CouchDB adaptor that works in serverless mode. But (as always) it's tricky to figure out just what's needed, and where and how to put it in. Hopefully Jeremy Ruston will take an interest (the CouchDB aspect alone would be valuable to TW) and provide some tips. (I have a query posted to their dev forum.) If it proves too gnarly I'll bail, we'll see.
Mar 3, 2014 at 6:38 PM
AAAHHH!!! I see! What you are saying is that they use Node.Js FOR THEIR BUILD ENVIRONMENT!!! PouchDB, btw, does exactly the same thing. You can build different versions of PouchDB for different environments and they use Node.Js to drive the 'build' process of taking all their components .js files and combining them together into one 'big' file and then minifying it and providing the final release.

I thought you were saying they used Node.js at run time to actually provide server functionality (something PouchDB does, e.g. PouchDB uses Node.Js both as its build environment and as its server infrastructure at run time).

Using Node.JS for builds isn't any kind of problem.
Mar 3, 2014 at 7:12 PM
I've replied to Jon over on the TiddlyWiki discussion group, but wanted to pick up a couple of points here. I'm very interested to help you get this running.
AAHHH!!! I see! What you are saying is that they use Node.Js FOR THEIR BUILD ENVIRONMENT!!!
TW is fairly mind bending in this respect. The Node.js code that we use to build the index.html file that you see at http://tiddlywiki.com is functionally equivalent to using something like Grunt to concatenate fragments of JS and HTML. But, it is implemented using the parsing/rendering engine at the heart of TW. Building the TW index.html file on the server is actually a matter of parsing and rendering a tiddler that in turn transcludes all the components that are needed. The standard template is:

https://github.com/Jermolene/TiddlyWiki5/blob/master/core/templates/tiddlywiki5.html.tid

When TW5 is being used as an HTTP server it goes through the same process to create the initial HTML file that is served to browsers.

Saving changes in the browser again just renders the same template tiddler to create the new HTML file.

A slightly different template can be used to generate an entirely static representation of the same tiddlers:

https://github.com/Jermolene/TiddlyWiki5/blob/master/core/templates/static.template.html.tid

It may go without saying that I've always loved the way that FORTH folds in on itself, reusing the same simple mechanisms to build the basic plumbing and to provide a user interface. In the same way TiddlyWiki endlessly re-uses the same basic mechanisms (for example, transclusion) to implement a wide variety of apparently unrelated features: for example, the translateable UI and user defined colour palettes are both really just applications of transclusion.
I thought you were saying they used Node.js at run time to actually provide server functionality
TW5 does indeed do just that.
Mar 3, 2014 at 10:36 PM
Jeremy, thanks for joining the conversation!

I read through the docs and I'm glad to see that TiddlyWiki can run without Node since, as explained above, our Node story for Android sucks.

In fact I looked specifically at TiddlyFox and it seems to me that porting that to Java should be trivial and as a result we could host a TiddlyWiki in a WebView with Thali (read: CouchDB) as the back end and it should 'just work'.

The big issue for me is - how do we handle conflicts? In other words, imagine that you (Jeremy) and I are both running our own local copies of tiddlywiki.com and we both edited the same entry and we both synch with each other and now we have a conflict. CouchDB will handle this for us if we let it, it will just pick a winner. But that can be incredibly confusing (where did my changes go?!?!?). Now yes, we have scenarios where this isn't an issue. Where the TiddlyWiki is really a blog and where all we really need is comments not conflict resolution. But I would love to move our wiki which is running on MediaWiki to TiddlyWiki. And for that we need some kind of conflict resolution.

Just curious,
 Yaron
Mar 5, 2014 at 11:08 AM
Hi Yaron
In fact I looked specifically at TiddlyFox and it seems to me that porting that to Java should be trivial and as a result we could host a TiddlyWiki in a WebView with Thali (read: CouchDB) as the back end and it should 'just work'.
Yes, your Android host app should be able to emulate the TiddlyFox DOM interface. The result will be that you'll be saving the entire TiddlyWiki document in one gulp.

The alternative would be to use the sync mechanism, where individual tiddlers are synced with the server as they change.

One potential advantage of the sync approach is that edit conflicts will be much rarer, and easier to resolve. A UI for resolving edit conflicts is in any case required as part of the TiddlyWeb adaptor.
Mar 5, 2014 at 1:40 PM
Edited Mar 5, 2014 at 3:31 PM
Yes, I'd much rather use the sync approach. Granular per-tiddler storage makes the best use of CouchDB. I think I'm closing in on a solution, after considerable yak shaving which I'll document over on the TiddlyWikiDev forum.

Hmm. Since Google Groups doesn't offer permalinks to individual posts, I'll crosspost here.
Mar 5, 2014 at 3:29 PM
Edited Mar 5, 2014 at 3:30 PM
Here's a status report, I think I'm on the right path but want to document where I'm at in case not.

I've built the tank edition, with a CouchDBAdaptor that was initially a clone of TiddlyWebAdaptor. The resulting app.html is loaded into a CouchApp as an attachment, which makes it available initially at an URL like http://localhost:5984/tiddly/_design/tiddly/app.html. Of course that immediately fails because the browser app makes server requests for things like status and tiddlers.json. So I am separately running the tw5.com-server edition (in a node.js debugger), and selectively pointing the browser app's CouchDBAdaptor at that server. So for example the failing request localhost:5984/tiddly/_design/tiddly/status becomes localhost:8080/status which succeeds.

(It didn't succeed initially, because of cross-browser constraints, but I added "Access-Control-Allow-Origin" : "*" to the relevant handler in modules/commands/server.js so that it would.)

Next I captured the server's responses to GET http://localhost:8080/status and GET http://localhost:8080/recipes/default/tiddlers.json, added those to CouchDB as attachments, and adjusted CouchDBAdaptor to fetch those things from CouchDB rather than tw5.com-server.

So now we have a CouchApp-style TiddlyWiki which is doing "browser-sync" not "server-sync" and can (I hope) incrementally rely less on tw5.com-server and more on CouchDB.

The next step was to trigger sync. This happens when I edit the GettingStarted page. And here it wasn't enough to point CouchDBAdaptor.prototype.saveTiddler at the tw5.com-server. When I did that and triggered sync, the browser's GET request mysteriously became an OPTIONS request which is unsupported in modules/commands/server.js. What's up with that? More CORS voodoo. As per http://metajack.im/2010/01/19/crossdomain-ajax-for-xmpp-http-binding-made-easy/, I added this to server.js:
this.server.addRoute({
    method: "OPTIONS",
    path: /^\/recipes\/default\/tiddlers\/(.+)$/,
    handler: function(request,response,state) {
        response.writeHead(200, "OK",
        {
        "Access-Control-Allow-Origin" : "*",
        "Access-Control-Allow-Methods" : "GET, POST, OPTIONS",
        "Access-Control-Allow-Headers" : "Content-Type",
        "Access-Control-Max-Age" : "86400"
        });
        response.end();
    }
});
And that worked.

Meanwhile, I add the following to the app.html generated by tankbld:
<script src="pouchdb-nightly.js"></script>

<script>
var global_couchdb = new PouchDB('http://localhost:5984/tiddly/');
</script>
(Note, it is also possible to do this:
<script src="/_utils/script/jquery.js"></script>
<script src="/_utils/script/jquery.couch.js"></script>

<script>
var global_couchdb = $.couch.db('tiddly');
</script>
However PouchDB is much nicer than CouchDB's jQuery plugin.)

Here's a test doc I placed into CouchDB using Futon (CouchDB's browser-based admin app):

curl http://127.0.0.1:5984/tiddly/103239ecc89696cb1407737967000884

{"_id":"103239ecc89696cb1407737967000884","_rev":"1-6556310e90b1c5dcffa5d89e6291
99dc","test":"jon"}

In the TiddlyWiki CouchApp's debug console we can do this:

global_couchdb.get('103239ecc89696cb1407737967000884', function(e, r) {console.log(r)} )
XHR finished loading: "http://localhost:5984/tiddly/103239ecc89696cb1407737967000884?_nonce=Xnt66eg6t5DKroQv". pouchdb-nightly.js:4189
Object {_id: "103239ecc89696cb1407737967000884", _rev: "1-6556310e90b1c5dcffa5d89e629199dc", test: "jon"}

Like I said, this is all just yak shaving to get to a place where I can observe the TiddlyWeb conversation from a CouchApp point of view, and reason about how to proceed.

And here's where you can likely advise.

I can see, for example, that the CouchApp is polling tiddlers.json. Presumably it's looking for changes? At the moment it won't find any, because that is only a static doc. I can also see that when the browser app saves to the server, the server updates its own instance of the wiki. Presumably those updates are then reflected in the next tiddlers.json sent back to the browser?

My best guess at this point is that I now need to:
  • at startup, externalize all the tiddlers in the app.html image as CouchDB docs
  • make CouchDBAdaptor's saveTiddler save to CouchDB and also update the CouchDB instance of tiddlers.json
  • redirect CouchDBAdaptor's loadTiddler and deleteTiddler to CouchDB
That's the plan, but please let me know what concepts I'm still missing. Also I'd love to hear what strategies you use to debug, especially in the browser. The way in which modules are loaded makes it an interesting challenge to debug!

Thanks!

Jon