Getting to our milestones

Feb 27, 2014 at 5:04 AM
I was hoping to get a sense of our milestones and their likely time lines by today but today was a disaster of meetings and I simultaneously severely underestimated how much time it would take just to lay out what we want to do for the next milestone. So I'll keep working on it tomorrow.

But I did make a bunch of progress on fleshing out the issues and getting them organized by milestone. See for the links.

My strong guess is that we want a release cadence of about 2 months between 'major' releases. The amount of work in milestone 2 is way, way beyond that. I haven't finished the estimates but even an eye ball glance makes that clear.

My main thought is that our next milestone could just be for dedicated apps which use Thali as an internal component (e.g. no external Thali Device Hub) in ad-hoc networking and peer-to-peer scenarios. This actually hits our two tops scenarios but it doesn't provide any of what I think of as the Thali super powers in terms of externalizing data. But I do believe in crawl, walk, run. And I think this is something we might be able to get done in 2 months or so.

In any case I'll keep working on this tomorrow and see if I can't get things organized in some sane way. But whatever we do we really need to get partners signed up for the scenarios the milestones are powering or we are just kidding ourselves.

Thoughts are heavily encouraged. So please share 'em!

Feb 27, 2014 at 9:53 PM
I just spent an unconscionable number of hours doing scheduling stuff. I was trying to flesh out our work items and get a ball park on their sizing. Originally we were broken into two milestones. The first would just have the functionality needed to write a Thali app so devs could kick the tires and the second would have the items needed to make this thing usable by humans.

My ball park estimate for the first milestone was 6 calendar months for 1 person.

We aren't going to go without a release for that long and anyway that's too long to plan out anyway for a project like this where we are constantly learning.

So I talked with Jon and we agreed on the following milestones:
Milestone 2.1 - Supporting CouchDB Standalone Applications - This puts in support for writing Thali applications as CouchDB Standalone applications. These are HTML/JS/CSS apps that are stored as attachments in CouchDB. Note, btw, that we already support native apps in Java/Android and .net.

(2L * 5 Dev Days + 6S * 2 Dev Days) / 4 Dev Days/Week * 1.5 Buffer / 4 Weeks/Month = 2 Months

The previous estimate is 'core development'. Jon is also signed up to provide several demos of apps written using the CouchDB Standalone Application support.

Milestone 2.2 - Supporting Tor both for outgoing connections and running Thali Device Hub as a hidden service on both Android and Java

2L * 5 Dev Days / 4 Dev Days/Week * 1.5 Buffer / 4 Weeks/Month = 1 Month

Again, the previous is just 'core development'. Jon is signed up to provide an app that leverages the Tor support.

The problem with both of these milestones is that neither actually lights up any scenarios on its own. They are 'table stakes' for other features that we need for scenarios. One of our goals then during these milestones is to get agreement on what specific scenarios we want to light up and in what order. See here for our canonical scenarios.
Mar 3, 2014 at 5:30 PM
I wanted to break down a bit more what we are up to.

From above:

Milestone 2.1 - WebBridge - 2 Months
Milestone 2.2 - Tor - 1 Month

Now let's talk about the Milestone 2 backlog.

QRCode Support - 1L + 1S - 1 Month
Hook up JVM version of Thali - 1L - 1/2 Month
ACL Infrastructure - 1/2 Month
Non-Browser (including webbridge) app registration for both Android and Java - 2L - 1 Month
Address Book/Group Management - 4S - 3/4 Month
Ad-Hoc Mobile Discovery - 1L - 1/2 Month

Note that these numbers are rounded so the backlog actually adds up to a bit less than 4.5 Months but that is the right ball park.
Mar 24, 2014 at 4:03 PM
So our next milestone is to release some code people can actually play with! We had previously released code that people could use to run PouchDB based Javascript code in a HTML app inside of Chrome and talk to Thali. But setting up this code was not easy (think registry settings) and it only worked if you had an Android device.

So what we are working on now, with a tentative release date of 4/11, is the following:
  • A version of the Thali Device Hub that runs on Java! (Massive thanks to the CouchBase folks for releasing the driver we need to make this happen)
  • An adapter for PouchDB that supports Thali via PouchDB's Javascript API. (Massive thanks to the PouchDB folks who accepted all the pull requests needed to make this happen into their code base!)
  • A WebView container that you can pour your own HTML/JS/CSS/PouchDB code into and have it run in the Thali context and talk to the Thali Device Hub
  • A demo based on the hackfest we did a few weeks ago at Microsoft.
Note that the code we intend to release will work on Windows and on Android. The only thing keeping it from working on Mac or Linux is that the CouchBase folks haven't yet built their driver Jar build.gradle file to cross compile platforms. So when you run a build, at the moment, you have to pick one platform. This is a bug and it will, I'm confident, eventually be fixed. If anyone needs to run things on Linux or Mac let me know and I can pretty easily whip up the Jar you need.

Now, just to set expectations. This is still test code. It is not properly tested and the security infrastructure is just a place holder. Our intention is to roll out updates to this code every few weeks or so to put in the missing pieces (like how to securely exchange identities via QR Codes, how to insecurely exchange identities via email, support for local discovery via mDNS or SSDP, a proper ACL engine, Tor support, etc.).

In any case expect more information about this written by people who are actual masters of the English language as we move forward.
May 6, 2014 at 3:25 AM
Edited May 6, 2014 at 2:26 PM
Right now the Thali Device Hub runs happily on Android and Windows.

Our demo code is based on PPNET which runs in Javascript using PouchDB. That works well for us because we are generally positioning PouchDB as our primary platform. So all good.

We have a good web client hosting solution for Android KitKat where we have a phonegap/Cordova style framework that takes our PPNET demo code (remember, all HTML/JS/PouchDB) and wraps it up as an Android app.

Where things aren't so good is on the PC for the client. We have a super fragile extension for Chrome where we can run the demo. But we really want code we can share with other people and the Chrome extension is too fragile to share.

Because it was easy I've been looking into what it would take to support JavaFX 8.0's WebView. The problem there is that their WebView code while nominally HTML 5 is really old and missing basic features like IndexedDB and WebSQL. This is a problem because the PPNET demo needs a local PouchDB instance.

Now I thought we had some good news because the latest release of PouchDB includes an IndexedDB polyfill that runs on HTML 5 local storage which Java FX 8.0's WebView supports.

I spent today testing this functionality out and the results weren't pretty. The really good news is that I fully expect PouchDB to succeed in using the polyfill. The really bad news is that it's clear that the PouchDB folks have literally never tested this code on a platform like JavaFX. When I realized that even the validate function didn't work (e.g. the thing that tells the adapter if it can turn on) I knew we were in trouble. I've run into other issues like problems with allDocs and then(). There are also issues with PouchDB's eventing framework on JavaFX.

Now the good news is that all these bugs are really minor. So again, the probability of success is good. But the bad news is that we are going to be their first customers on a framework the PouchDB folks have been very clear is experimental. I'm kind of used to that (anyone remember the early days with Couchbase?) but here's the thing - We really don't want to run on JavaFX.

JavaFX has a dated WebView and no plans I've seen to update it. Furthermore nobody in any of the communities we travel in seem to be using it. So we are really out on our own. It seems much more reasonable for us to invest in something like Chromium as our PC based execution environment. Lots of people use Chromium and this would mean that our web code was reasonably consistent across both Android and the PC. But unfortunately Chromium is in C++ and we use Java so we would have to use the Chromium Extension Framework to get access to Chromium and looking at the bugs it's clear that the Java Chromium Extension Framework is still in its very early days.

And did I mention we want to do a demo at the end of the month?!?!? We are supposed to be at HTML 5 in SF. I really don't like the idea of taking a completely new platform I've never touched before that uses JNI to boot and getting that ready by the end of the month.

So we have a couple of options that seem reasonable to me:
Android Only - This was Jon's extremely reasonable idea. Everything works client side on Android. So we could just do the demo using two Android devices. This makes me sad because we have done a ton of work to get things working on the PC and I really want people to understand that Thali is more than just a mobile play. But of all the options this is by far the least risky as it uses PPNET as is.

JavaFX - We could choose to be first in and just go the JavaFX route. This works well with the Android Only option as, if we get it working, the code should 'just work'.

Modify PPNET - Right now PPNET runs by using local PouchDB to talk to a remote server. It's beyond easy to throw in a Hub as the server, the problem is the local PouchDB. Local PouchDB works like a champ on Android but as explained ad nauseum above it doesn't work very well on JavaFX. But it wouldn't be brain surgery to re-write PPNET to exclusively use the remote hub and not use local PouchDB at all. In return for this our code should run just fine on JavaFX and Android. The only real work items here are maybe fixing the blob interface in XMLHTTPRequest (it currently supports strings, not blobs but we can fix that) and putting in Javascript View support in the hub. These are both things we should do anyway and should be relatively low risk (although the Javascript View support makes me nervous but not super nervous). What I don't like about this option is that it's artificial. In the real world nobody would want to run exclusively via the hub. They would want to have data inside their app if they can for performance reasons. But still, the hub would be local so this isn't too bad.

Dual head on localhost - This idea is a variant on this. The idea is that we can run in a completely normal browser (at least on a PC). The way things would work is that we would put the app in a directory on the file system. The user would call a Java app inside of that directory. That Java app would generate a bearer token and save it in a .js file in the directory with the app (which means no global installs but I can think of ways around that limitation in the long run) and then call the default browser with a file URL pointing to the index.html file in the directory. This would launch the browser who would get the bearer token by doing a relative reference to the .js file in the same directory. All the browsers, I believe, will allow this. The browser would then load the index.html which would be completely normal HTML. No magic. No Java bridge. The only interesting thing it would do is pull in the bearer token from the local .js file. We would then put a second head on the Thali Device Hub to listen to local host. It would also expose the CORS headers so we wouldn't have cross domain issues. It would only honor requests that have an authorization header that contains an authentic bearer token. I'm using a header instead of the URL (as I did in the proposal linked above) because we are using a normal browser and one should never, ever, put a bearer token in a URL that is going anywhere near a normal browser. This means that the app can only talk to the Thali Device Hub via PouchDB (where we would set the authorization header as a config option). The upside is that this would work with the local browser. The downside is that it only allows the local browser to talk to the local Thali Device Hub and nothing else. So, for example, the current PPNET demo takes a shortcut and uses a single Thali Device Hub for both devices. That won't work in this scenario. Well, sorta. The Android client could use our existing infrastructure so if we ran the single Thali Device Hub on the PC then the Android client could come in through the mutual SSL auth head (which is network accessible) and the browser based client would come in through localhost. So yeah, I guess it would just work.

Try for Chromium - This is really a variant on the Android Only option. We would assume that we would only demo on Android but I would spend my time getting Chromium up and running. If I can get it going fast enough then we would demo on the PC with Chromium. From the demo perspective literally not one line of code should have to change between Android and Chromium so it either works or it doesn't. What I like about this option is that I suspect we want to end up in Chromium in general. JavaFX's WebView just isn't getting any love. What worries me about this option is that we have another demo coming up in June and in order to get that demo I need to have QRCode support. In theory we have enough time to get both Chromium and QRCode support for June but it would be tight and it could threaten our demo there. Now, to be fair, the worst case we will be in London. It's just that we might get a key note if we have QRCode support and we won't get the keynote if we don't. But, to be fair, we might not get the keynote even if we have QR Code support. So there you go. :) Do note that JCEF (the Java Chrome Extension Framework) is not exactly super solid. Looking at the bugs it's clear that the thing isn't exactly production ready and it doesn't present itself as being so. Which makes me wonder if it's the right dependency to take. But if we want a web hosting solution that can use our Java code on the PC we don't have many options.

Package up Chrome - Our Chrome extension actually works reasonably well. The main problem with it is that it needs an installer that needs to do things like mess with the registry (this is required by Chrome). Also the thing is a huge stinking security hole and it would take real work to get it to package up in a non-insane way. Basically we would have to create an installer installer (yes, seriously) to create custom installers for each app so they can all create their own separate security domains. And then we can only distribute the installer via Google's store. There is no side loading. Which is not very peer to peer friendly.

May 6, 2014 at 3:02 PM
My $0.02:
  1. I want to add as little new risk as possible because there's still old risk on the table: continuous sync, TJWS, etc. We need to at least know what's going on there, now we don't, this warrants near-term effort.
  2. The Android-only scenario doesn't need to be Android-only. I.e. we could do Android on a tablet + Android on a PC (in the emulator) + something else.
  3. The Chrome extension is the currently known-to-work something else. It's also the premiere debugging environment, since we've seen that chrome:inspect can destabilize adb when talking to the emulator, and in any case is not full-featured w/respect to standalone Chrome.
The Chrome extension can be a demo-only fallback in the short term, and a dev-only thing in the long term.

Short-term, we'd use it for SF if a) we are solid w/respect to continuous sync, but b) that didn't leave enough time for JavaFX or Chromium. (But remember it's a third leg of a stool: Android 1 + Android 2 + Chrome.)

Long-term, it's something devs may well want (I sure would) enough to work through the (gnarly but not insurmountable) setup. (Which we can perhaps smooth with a bit of automation.) (And if the .NET bridge goes away in favor of a generic Java bridge that story improves.) So: real Thali apps never use the extension, installers and the store are not issues, it is just a helpful tool in the kit.
May 7, 2014 at 1:01 AM
Jon and I discussed this and reached a few conclusion.

We are two weeks before the HTML 5 demo. Now is NOT the time to play around with the code.

So for HTML5 we will use Chrome and Android and call it a day.

I will focus on getting our code updated to the latest Couchbase releases and seeing if the problems we have run into with connection exhaustion and replication failures repeat themselves.

Once I am confident we have good bits for the demo then I will go work on the dual headed localhost solution. It will let us run our code in any browser and get rid of the Chrome extension dependency we currently have.
Jun 26, 2014 at 3:24 PM
Edited Jul 10, 2014 at 6:30 PM
I wanted to revisit the list of features we need before we are 'feature complete'.
  • HTML5 app development support
  • TOR support
  • QR Code support
  • Global address book
  • an ACL engine,
  • an ACL UX,
  • a group UX,
  • a data clean up/cache management/storage quota management service/bandwidth quota management (e.g. to delete data the user doesn't need when they are low on disk and to keep incoming writes from taking up too much bandwidth/disk space),
  • a queue based message delivery service - this would be used for sending the equivalent of emails, calendar invitations, etc. This is different than synchronization in that these are one time events that need to be reliably sent, not an ongoing synchronization.
  • a central replication manager,
  • a battery management mechanism for the replication manager,
  • a mechanism to evergreen the TDH (e.g. automatically update it),
  • a last update wins replicator,
  • make couchbase's query capabilities compliant with CouchDB,
  • a way to securely register apps with the TDH and
  • a way to let apps run without the TDH but then agree on a TDH if it's there
  • replace TJWS in couchbase listener with Apache Listener or something equivalent
  • support registering link handlers to let users click links in places like email to add a user to their address book
  • Extend address book UX to support sending emails with invitations to connect
  • a mechanism to let apps start the local TDH in case it's not already running
  • ad-hoc networking support
The first four are actually almost done and once done will allow for app development in Thali as we flesh out everything else. I expect to see those in depot in the next week or three.

The rest aren't brain surgery but they will take time to suss out. I suspect we'll need to take an iterative approach where we first put in 'something' and then improve it.

But in any case this gives some sense of how much work there is to be done to get to feature complete.