Weekly Status

Feb 24, 2014 at 9:51 PM
Edited Feb 24, 2014 at 9:53 PM
Last Week
  • Spent a ton of time with Jon helping him to get going and generally conspiring with him on where Thali should be going. Expect to see our thinking put out on this discussion group.
  • As part of a really dorky pull request to CouchBase I rewrote our instructions for pull requests.
  • Filed issues with LoveSeat and PouchDB to put in features Thali needs.
  • Synch'd with the changes in CouchBase for the last month (including the Java refactoring) to our CouchBase forks. Also started moving code around in the Thali repos to take advantage of the CouchBase refactoring.
This Week
  • Finish fixing up Thali's repos with the CouchBase changes
    • This includes refactoring code, fixing dependencies, etc.
  • Finish testing CouchBase replication
    • We have run into enough issues that we have to test the core functionality
  • Start adding Tor support
Mar 3, 2014 at 6:58 PM
Last Week
  • TL;DR - Working on schedules and getting us up and running with CouchBase's latest
  • Spent two days getting Thali's depo fully synch'd with lastest changes from CouchBase. Including moving all our test and functional code out of Android and into Universal! Note that I still haven't finished moving the TDH code.
  • I lost most of Wednesday to random meetings.
  • I spent a chunk of Wednesday and most of Thursday on fleshing out our work items and schedule.
  • I spent a bit of Thursday and all of Friday on Issue 2 - getting pull support to work right with CouchBase Lite.
This Week
Mar 11, 2014 at 1:18 AM
Last Week
  • TL;DR - Bug hell, depo cleanup, started working on Enabling XMLHTTPRequest in WebView
  • Monday was spent with a cascade of bugs
    I kept getting weird null pointer exceptions which turned out to be a change in CouchBase's code in one depot but they had never updated their maven repository to update another depot that had a dependency on the first. Figuring that out took a while.
    Fixing this required me to build everything which triggered a bizarre Intellij bug which masked a Gradle version mismatch as a maven error.
    Which, once I figured that out, led to another Gradle bug. In 1.10 they 'fixed' their support for mavenLocal() but apparently NOT for AAR files. Thankfully I knew the fix but it a took a while to figure out the problem.
  • Tuesday was spent in replication issues
    I wasted several hours due to a screw up on my part. When I had merged in some changes the updates from CouchBase overwrote a fix I had for a bug in their code. I left myself a note to follow up and put the fix back in. Forgot about the note while testing other things. Got other issues cleaned up and spent a fair chunk of the day finding the same bug I forgot to put in the fix for.
    I also spent a good bit of the day explaining this bug in detail to the CouchBase guys which thankfully worked since they put the fix for it into their depot.
    I then spent the last part of the day tracking down a bug in pull replication which turned out to actually be me misreading why a break point was triggering. :(
  • Wednesday I was sick so I did email, filed bugs, closed down issue 2 and started to get GitHub and CodePlex updated with our latest changes and start to update the instructions for how to build Thali.
  • Thursday I finished updating the depos and the related instructions and also got the Java Utilities working again, a piece of work left over from the great merge some weeks ago.
  • Friday I started hacking on the JS code mostly to get the XMLHTTPRequest code cleaned up per issue 26
Next Week
Mar 18, 2014 at 12:17 AM
Last Week
  • TL;DR - An endless mess of debugging that eventually led to the PouchDB + Thali code working correctly in Chrome
  • Monday - Spent some time trying to think about how to solve the problem with PouchDB mangling urls and testing my solution. I then had to fix up the .net code in the Chrome host to work with the new design. I spent some time getting Jon up and running with our Chrome host. I then had to spend some time on some build issues that were causing problems. And then I rewrote our provisioning code in .net so it happens explicitly at the start and now all urls except at provisioning are 'full' httpkey urls! So, busy day.
  • Tuesday - I ended up getting sucked into a bunch of meetings but most of the day was eaten up trying to get replication working in PouchDB. I lost the first half of the day to a PouchDB bug where it turns out that you have to submit arguments to the ajax adapter in PouchDB two different ways. One way via normal options and another way via a buried options argument inside of options. It turns out Ajax options get submitted one way to 'destroy' and another way to the general adapter. But I didn't know that and it took a bunch of debugging to figure it out. The rest of the day was lost to a Javascript constructor issue. It turns out, silly boy that I am, when I tried to create an adapter I just called 'new'. But the adapters depend both on 'new' AND on a custom 'this' pointer. I didn't realize the second part until after a lot of debugging and only then realized I had to use Javascript 'call' in order to pass in a custom this pointer. So I spent most of the day on what turned out to be about two lines (seriously) of Javascript code to fix. The reason for all the pain, btw, is simply that Javascript is deep stacks of asynchronous callbacks, often in contexts where I can't directly step into the function being called. So I repeatedly had to guess what function was being called, re-run the test with a stop point and hope I hit it. Needless, to say, not a productivity enhancing experience.
  • Wednesday - I then ran into an issue with CouchBase and replication that it wasn't handling the protocol correctly. So I patched that in my code and then edited the Javascript tests to run the test twice, once just using PouchDB as a database locally and a second time using the Thali Hub exclusively for storage. This then ran into an issue with how I was taking over getHost in PouchDB. I had my own hack but I knew it was bad and the PouchDB folks had suggested a cleaner solution so I had to rewrite the code to use the cleaner solution. But the problem was that in order to create a fix that PouchDB could take I needed to clone their depo, run their build and run their tests. I ran into a bunch of problems making that work on Windows.
  • Thursday - So I managed to at least get PouchDB to build on Windows and tried to make getHost public, which was the suggestion the PouchDB folks made. This didn't work for me at first. This led me to have to do a deep dive into the Javascript context inheritance model to understand why my attempt to make getHost 'public' was failing to properly inherit overrides. Meanwhile I thought (I was wrong) that my problem was that PouchDB wasn't building correctly. To be fair I was getting a ton of errors in Windows, but they turned out not to matter. But in the meantime I figured Windows wasn't going to work because something in PouchDB's node.js build process seemed to be failing and tests in particular just wouldn't run. So I ended up getting out VirtualBox and installing Elementary OS (an Ubuntu based distro) and installing Node.JS/Git/etc. there and cloning PouchDB there. The good news is that tests now ran there! In the end I figured out that I COULD do the build of PouchDB on Windows but I never figured out how to run tests so having the Linux distro running was useful.
  • Friday - Since I got getHost working properly yesterday I was now back to debugging replication. I kept getting failures! It turned out that there was a bug in PouchDB that was again mangling URLs. Finding that took a little bit but fixing it was easy. The reason finding the bug took a while is that PouchDB was 'sucking' up errors and disappearing them so my code would just stop and I had to guess why! Arg!!!! I then wasting a bunch of time trying to figure out why my custom adapter kept disappearing which turned out to be a screw up on my part where I had merged some code badly. :( I then ran into an issue that replications seemed to be oddly failing. This turned out to be another bug in CouchBase Lite. I worked around that one and ran into a problem I had seen before, the code was running glacially slowly! A GET request could take literally 20 seconds! I did a bunch of perf debugging and the issue is somewhere between the .net HTTP stack, adb and TJWS's HTTP stack. Since most of that won't be relevant for the demo I'm working on I'm going to shelve the perf work for the moment.
Next Week
  • TL;DR - Take the new CouchBase code and get Thali working in pure Java!!!!
  • Merge my xmlhttp code/chrome host work into main branch and hand to Jon to use for our demo work.
  • Run tests, fix anything broken and submit a pull request to PouchDB with changes
  • CouchBase delivered the first release of the Java code! I expect to spend the bulk of the week compiling it and getting it to work with Thali.
  • If by some miracle I finish up early then start working on enabling PouchDB in WebView in Java
Mar 24, 2014 at 6:41 PM
This Week
  • TL;DR - PouchDB - 6 Pull Requests accepted, CouchBase - 22 issues filed. Updated our Android and Java code to run with latest drivers from CouchBase.
  • Monday - Usually Monday avalanche of email. The majority of the day was spent putting together a single big pull request(PR) for PouchDB and making sure all the PouchDB tests passed. And opened a new bug against PouchDB.
  • Tuesday - I ended up making a copy of Thali on GitHub so I could point to individual line numbers in my conversation with the PouchDB folks about my PR. The PouchDB folks did not like my big PR so I had to re-submit it as 4 different PRs. I started to try to build CouchBase for Java! This led to a litany of bug fixes in their code.
  • Wednesday - Mostly hacking through numerous issues with the Java code. Ending up filling 12 issues with CouchBase capturing all the fixes I had to do yesterday and today.
  • Thursday - The CouchBase native code couldn't even theoretically work on Windows because it assumes that it only needs to load a single DLL and because they are using mingw they need to load 2. So I had to rewrite their code to deal with that. I then synch'd and it turned out they changed the signature of their core constructor and so I had to go through all my code to deal with that. Only filed 6 issues against CouchBase today.
  • Friday - Most of my PR requests to PouchDB were accepted except the most important one. This led to a long conversation and a request from the PouchDB folks to change how the code worked. Their request made sense so I changed the code, updated my dependent code, tested everything everywhere and submitted an updated PR. Which they accepted!!!! I worked on CouchBase where after a synch everything stopped working but thankfully they pushed out another update that fixed things. This then led to me running into a nasty bug in the Java SSL stack that broke my code but stackoverflow (bless their hearts) had this issue and the fix was beyond trivial. I ran into 4 more issues to file against CouchBase. And then I ran into the issue of death that ended my week on a sad note. The Java code from CouchBase doesn't handle nested sqlite transactions which wasn't supposed to matter because CouchBase wasn't supposed to use them. But it turns out it is and now I have to figure that mess out.
Next Week
  • TL;DR- Try to get Java version of CouchBase actually working and then start work on WebView support
  • I am really concerned about the sqlite bug I mentioned above. This is an area I know nothing about so I'll have to learn the basics. I have no idea how long this is going to take to untangle.
  • If I manage to get through the previous then it's time to work on enabling WebView to talk to our PouchDB adapter, this mostly involves writing a HTTP Proxy in Java.
Apr 1, 2014 at 1:49 AM
This Week
  • TL;DR - THALI NOW RUNS ON GENERIC JAVA!!!!!!
  • Monday - Tried to figure out last week's SqlIte bug. Turns out that CouchBase DOES need nested transactions for replication and the current CouchBase Java (not Android) code doesn't support that. I worked around the problem by editing CouchBase's Sqlite code to use SavePoints which are Sqlite's equivalent to Google's Android nested transactions. I then ran into a weird problem where my tests seemed to all fail due to a timing bug. I think the timing bug has something to do with awful perf from TJWS.
  • Tuesday - I investigated TJWS's awful perf. The conclusion of the previous thread seems to be that TJWS has awful perf but I do have a work around for now. This then led me to wonder why most of my tests passed separately but none together! This led to some ugly worries about how I've been merging my code with CouchBase's which led me to walking through all my forks of CouchBase and marking each line of code with the issue it came from. This helps me do merges much better since I know why I made a change. I'm pretty sure my test issues are related to a screw up in how I merged replication code.
  • Wednesday - I finished tagging code, wrote a letter to help Simi out and wrote a bunch of mail to help Jon out. I finally fixed the replication issue (it was a bad merge, an extra line got put in that shouldn't have been there). So my tests are passing individually but not as a group! The problem appears to be in trying to shut down the Java in the server. This led me to understand that TJWS doesn't actually shut down when you tell it to shut down! Interestingly enough I'm pretty sure this whole issue was a distraction. I think the real issue is that when you have outstanding continuous replications they don't give up file handles when you try to delete a database they are replicating which causes my test set up code (which deletes all dbs) to fail. My work around is to create a new directory to hold the databases for each test.
  • Thursday - I finally got Java working only to have CouchBase drop a completely new native driver with a different solution for the nested transaction problem. I tested that and thankfully it appears to work. But then I tried running tests in Android to make sure all my hacking in Java hadn't broken anything and sure enough it had. I tracked the problem down to having stuck an important setup command in front of an assert. This assert was always run in Java but asserts by default don't run in Android. You have to send an ADB command to make that happen. Took me a bit to figure out the problem. I then went and actually wrote the Thali Device Hub for Java! (For those confused, the previous code was getting Thali Utilities for Java running) Getting TDH for Java running took me longer than I thought because of a bunch of silly issues but I managed to slay them.
  • Friday - Was a mishmash of meetings and trying to get my new laptop to run. Had a great call with CouchBase about how we could all play together.
Next Week
  • TL;DR - Enable creating Thali apps via a webview using HTML/JS/CSS + PouchDB
  • Finish setting up laptop
  • Basically write a simple Java HTTP proxy to proxy JSON requests from our PouchDB adapter (that we already have from our Chrome work) to Ektorp along with set up support. This is literally just the Java version of the work we already did in .net for Chrome.
Apr 8, 2014 at 1:41 AM
This Week
  • TL:DR - We can now host PouchDB using the Thali Adapter in KitKat!
  • Monday - Took me all day to install everything on my new laptop
  • Tuesday - Spent most of the day tracking down two delete bugs (1 & 2) in CouchBase. Also made the code handle TJWS lying about shutting down in Java.
  • Wednesday - Figured out how to get bash to tell me when I had unchecked in changes across our 9 depots. Got Gradle to build everything for us in one way. Started hacking on the XMLHTTPRequest bridge for Java.
  • Thursday - Wrote the code for the Java version of the XML HTTP Request bridge. But ran into a very ugly problem with the PouchDB not running right in JavaFX's WebView.
  • Friday - So it turns out that PouchDB's notification model doesn't work for some reason on JavaFX. I reported the bug to PouchDB and they confirmed and are trying to fix. But, it gets worse! JavaFX supports local storage but not indexed DB or WebSQL. So Jon and I talked and agreed I would switch to figuring out how to get PouchDB with the Thali adapter running on Android.
    • This then led to me running into an old Android bugs where it gets conflicts on certain META-INF text files. The best part is that it took me forever to fix this because I had a conflict on TWO files and when I fixed the first one I would get the error for the second that looked identical to the first and not realize that there were two files in conflict instead of one. Eventually I figured it out.
    • Which led right into another pit, Android doesn't support the Jar URL so I couldn't load my resource files in the WebView (this actually works in JavaFX's WebView). I tried to implement my own intercepter for the jar URL type but this led to weird behavior and strange errors. In the end it (after wasting a lot of time) it turned out the errors I was seeing were spurious and the real issue is that Android API 18's WebView only supports a single WebSQL database per app (huh?!?!).
    • The solution to the WebSQL problem is to use Cordova, there is a plugin there that PouchDB supports. But I really didn't want to throw Cordova into the mix. So instead I decided to try Android API 19 (aka Android 4.4 aka KitKat) because it's WebView runs Chromium. So I downloaded the RC1 version of Android 4.4 from the awesome x86 project and it wouldn't run! I then tried to install the normal Android x86 emulator but it wouldn't load HAXE which is an Intel library needed to make it work. This led to a lot of investigation that eventually turned up that you can't run HAXE or VirtualBox and have Hyper-V installed. So I fixed that and got the x86 image running.
    • This then led to a problem with loading the HTML and JS resources I needed (remember the JAR URL issue?) which I solved in the most awful way possibly by copying the files (using a sure to break static list) out of resources and into local storage and then running the WebView from there.
    • At which point - IT WORKED!!!!!! So we can now build Thali apps using PouchDB in KitKat.
Next Week
  • TL;DR - Package up the Android work so Jon can use it and then get PouchDB with our adapter running in something on Java
Apr 22, 2014 at 12:17 AM
[Note: I was OOF last week!!! Vacation!!!!!!!]

Two Weeks Ago
  • TL;DR - Packaged up Android SDK work and identified a bunch of bugs and stuff I'll need to fix in replication in both PouchDB and in the TDH
  • Monday - Created new project for our Android Thali SDK (let's you write a HTML/JS/CSS app and have it wrapped up as an Android app with full Thali access), wasted entire day chasing down a bug that caused my databases to disappear. Turned out to be my fault, I had set a boolean wrong in a function I wrote. :(
    • So why didn't we just use Cordova? Because Cordova doesn't support the desktop and we want to use the same code for a desktop SDK. I hope Cordova gets fixed!
  • Tuesday - For some reason Gradle build refused to include assets (no clue) which I worked around by slamming in { main.assets.srcDirs = ['assets'] } into sourceSets in build.gradle. I also ran into an issue that I'm using Gradle to copy files around the copy doesn't happen early enough in the build process. I gave up on fixing this and instead just put in a fix in the universal build that runs the copy task before the build task. I spent some time trying to figure out why everything stopped working in WebView in Android. Turns out that in KitKat they changed the semantics of loadUrl for Javascript URLs to reset the whole page. The work around is to use the new executeJavascript function in KitKat. I then ran into an ocean of really bizarre behavior (listed below for those who are curious) that turned out to be that IntelliJ defaults new projects to Gradle 1.10 and I'm using Android Tools 0.9, which should work with 1.10 but clearly doesn't. Once I manually upgraded the project to Gradle 1.11 then everything worked fine.
    • Weird behavior included: if statements failing even when watches said it should succeed, call to functions that no one was calling, screwed up line numbers, classes.jar files shrinking between the AAR they came from and the exploded-aar they ended up in, etc.
  • Wednesday - Spent a bunch of time getting Jon running on the new code for the Friday demo. Then played around a bunch with JavaCEF and ran into issues like it not working over remote desktop. Fixed that by doing a build of their latest rather than using their last binary drop. Then it turns out they don't support inserting Javascript/Java calls. I eventually moved on to trying to do replication between two TDHs. This turned up a bunch of minor bugs that keep PouchDB from issuing a replication request on the server. This also turned up that PouchDB has a bug that keeps it from issuing remote server from replication requests correctly.
  • Thursday - Spent part of the day trying to figure out how to route replications from Android to the host. This turned into a silly chase of mindless solutions because I didn't realize that because I was running in VirtualBox I could just use the host's IP! Spent the day late into the night trying to get our Friday demo working with Jon.
  • Friday - We did our demo and I did some email and tried to recover from the late night hacking.
Next Week
  • TL;DR - Survive this week's demo!
  • The demo two weeks ago was actually in preparation for a demo this week but we have to put together slide decks, meet with people, etc. We won't be doing a public release until we get the Chromium SDK container running to match the Android SDK project. Realistically along with some replication bugs this is going to take at least two weeks.
Apr 28, 2014 at 6:24 PM
This Week
  • TL;DR- We survived the executive review demo and then I collapsed with a cold!
  • Monday - Digging out of a weeks worth of email from my vacation.
  • Tuesday - Wednesday - Getting ready for the executive review.
  • Thursday - Out of office being sick :(
  • Friday - Caught up on email (including ones I'd been putting off for weeks), did a full synch with Couchbase and started updating the Thali code to match the Couchbase updates.
Next Week
  • This is just my current backlog in prioritized order, I don't actually expect to get this all done this week:
  • Need to automate running tests against all our projects because I've found rotting code
  • Need to fix the issues we have with hub to hub replication (mainly what appears to be a bug in PouchDB)
  • Need to get JavaFX set up as a SDK the same way we did with Android
  • Once all that is done we will be ready for our first pre-alpha release
  • Then go put in QR Code Support along with some kind of address book - That will be our second release
  • Then put in TOR support - That will be our third release
May 5, 2014 at 6:34 PM
Edited May 5, 2014 at 6:40 PM
This Week
  • TL;DR- Got Couchbase Android to test against itself, got all of our tests back up and running and made progress fixing a replication bug in PouchDB.
  • Monday - We let our tests rot! So I started to automate running all our tests across all of our depots. Amongst other things I found out I had set up Couchbase Android's tests but I hadn't, I had configured them wrong! I then ran into other bugs around the fact that I had upgraded them to a newer version of Android and a bug in the Couchbase code base that Couchbase thankfully had a fix for.
  • Tuesday - Lots of phone calls in the morning. I then changed Couchbase Android's tests to use the listener instead of Sync Gateway so the tests could run locally. This ran into some bugs I fixed. I got just about everything running but a few tests were failing. I investigate one and discovered that there was no way that test could ever have been passing! This led me to discover that Couchbase isn't running Travis on checkins right now because of config issues. So I finally had to give up since I couldn't tell the difference between tests that Couchbase knew were bad from ones that went bad because of my switch to the listener. I'll come back to this later but for now I had to time box it and move on.
  • Wednesday - Lots of phone calls and emails in the morning. Turns out our UniversalUtilities tests weren't even building! I then spent a good chunk of the day trying to figure out why the JavaUtilities tests were failing and eventually discovered that Gradlew was running 64 bit Java but Couchbase's code on Windows only works in 32 bit Java. Sigh.
  • Thursday - I finally got all our tests passing and started to work on getting the hub to hub replication tests running for the first time. This led to the realization that the code would be unmaintainable (and likely un-writable) unless I used ES5's 'then' which while not supported by JavaFX (where I'm testing) is supported via polyfill by PouchDB. So I read up on how 'then' works and re-wrote the test to use PouchDB's version of 'then'. I then ran into weird behavior which turned out to be a bug that I thought PouchDB had fixed but hadn't, the old 'MutationObservers don't work right with PouchDB' bug from several weeks back. So I was able to apply a fix to that.
  • Friday - Confirmed that PouchDB has a bug in their replicate.from logic when replicating on the server. The problem is that I can't get PouchDB to build on Windows (yes, I filed a bug, yes it is being looked at). Since I needed to make progress this meant I got to spend a big chunk of the day setting up Elementary OS and running into tear inducing problems getting PouchDB tests running. All the issues with running PouchDB tests turned out to be related to Elementary OS (read: Ubuntu) loading really old versions of CouchDB, NPM, etc. that gave PouchDB's build and test infrastructure indigestion. I finally got the tests working. Fixing the actual bug was easy but now the tests don't pass with my fix. That isn't surprising since my fix changes their incorrect server side commands to the correct ones. So I'll have to see how I broke their tests.
Next Week
  • We have to figure out what will work in JavaFX and that will tell us what work we are doing
  • Figure out of PouchDB's polyfill for IndexedDB will work in JavaFX - The answer to this question will completely change the backlog ordering
  • Get PouchDB tests passing so I can submit a PR for the bug I fixed
  • Finish our hub to hub replication tests by setting them up to test local to hub to hub to local replication.
  • Get JavaFX set up as a SDK the same way we did with Android - First Public Release
  • Put in QR Code Support along with an address book of some kind - Second Public Release
  • Put in TOR support - Third Public Release
May 12, 2014 at 7:46 PM
This Week
  • TL;DR - Abandoned JavaFX 8.0 WebView. Automated Couchbase's tests so they run in both Android and Java automatically against Couchbase lite instead of a different platform. Got everything building right in Thali and did a full release for Jon to play with.
  • Monday - I tried to get PouchDB to build on Windows and to get PouchDB to run in JavaFX 8.0's WebView. I failed on both.
    • I ran into a whole nest of problems with PouchDB in JavaFX. The problem is that JavaFX doesn't support indexedDB or WebSQL. But PouchDB just released support for an adapter called localstorage-down that can implement an indexedDB style interfaces over HTML 5 local storage which JavaFX does support. I couldn't get anything running in JavaFX though so I separately set up and ran the localstorage-down smoke test. Thankfully that worked which indicated that there was hope.
    • False hope actually. It turns out yu have to build a different version of PouchDB to get it working with the level down adapters. I built the wrong one. I build level-js. This is actually a pass through adapter that just calls through to levelDB which isn't supported in HTML5 at all (it's used in Node.js). I eventually got on IRC where the PouchDB folks set me straight. I needed to build the localstorage-down custom version of PouchDB. Which I did and nothing worked. I tried to debug but JavaFX has no debugging infrastructure and Firebug lite can't even see the source of the Javascript files so I can't do step debugging! ARG!!!!!
    • Eventually I got the right version of PouchDB but nothing is working. It turns out that the problem is in PouchDB. They have a function called 'LevelPouch.valid()' that is hooked up when creating localstorage-down and it has a bug so it can't ever work on platforms (like JavaFX) that didn't have levelDB! What seems to have happened is that all the tests were run on a platform that did support LevelDB and so they didn't realize their validate function was bust. So I filed that bug and hard coded the function to be true. This immediately led to a really bizarre bug. Combine this with the generally bizarre state of JavaFX all up and I came to the conclusion that JavaFX is just not the right environment for us. I wrote an article exploring our options here
  • Tuesday - I updated our forks of Couchbase to the latest and greatest and immediately ran into problems with compiling the native libraries. After an enormous amount of debugging I eventually figured out that the problem had to do with Visual Studio not properly setting up my environment. All the gory details and work around are available here.
    • At this point I realized that there were no Java tests in Couchbase! It turns out that they have a 7 step process you have to run through in order to copy the Android tests over. So first I had to get the Android tests running before I could copy them over. Which I did. Then I followed the 7 steps but they were buggy so I filed a bug to fix them. Then I ran into bizarre problems with magical windows DLLs showing up in strange places which turned out to be my fault! I had needed the magical dll back when we were using mingw to compile on Windows but now we are using VS (per Couchbase's desire) and the dll isn't needed. So I wasted time figuring out my own mistake.
  • Wednesday - So now I can build native successfully but only in cmd not from git shell (where I normally work from) because the .bat file that VS uses to set things up won't run in the git shell. I also ripped out the magical class loading logic I had put into Couchbase to deal with mingw since we aren't dealing with mingw anymore. The rest of the day got lost in internal Microsoft stuff.
  • Thursday - The entire morning dissolved into an endless series of calls. I then spent time trying to figure out why the Couchbase tests were failing in Java. After much bizarre debugging including class resolution errors I decided to walk away from Java and instead put together a PR to Couchbase to make changes to their Android tests that would let us test against Couchbase Lite. That's right, Couchbase Lite isn't tested against Couchbase Lite! It's tested against Sync Gateway. I made some changes to fix that and sent in a PR for them. When I went back to the Java issues I had a hunch and completely nuked my intellij IDE files and re-imported from gradle. Sure enough the Java tests were more or less now working! It turns out that Intellij gets confused about which Jars to load sometimes. This then set me up to submit another PR request to Couchbase to automate their Java tests so we didn't have to do the 7 manual steps.
    • Meanwhile the PouchDB folks (thanks Calvin and Nolan!) have been working on the bug I filed on PouchDB not working in Windows. Calvin sent down a possible fix which I tested but alas it didn't work.
  • Friday - Sigh... so now I switched to getting Thali's tests running (yes, Thali, remember that?). But I was getting failures on our Javascript tests in Android. After much debugging it turns out I was running the wrong version of PouchDB. So I switched to the right version and suddenly PouchDB disappeared completely! Huh? After more debugging I was reminded why hacking is such an awful idea. It turns out that some time ago I ran into a nasty problem in Android that I couldn't get it to enumerate for me the files in a folder in a Jar. I can address the files directly if I know their names but if I don't I couldn't find anyway to list them! So I did something very stupid and now I paid for it. I hard coded in the names of the files I needed to load in Android. But of course the name for PouchDB was something like PouchDB-2.1.0.js and I updated to PouchDB-2.2.0.js which wasn't in the list! I knew when I put that code in that it would be fragile and I would pay and the payment came due.
    • Thankfully fixing it wasn't too hard. I created a Gradle task that generates a manifest file during build time with a list of the files in the Jar. Then I wrote code that reads in that manifest and gets the file names and loads them. That is the kind of hack that I can live with for a while. Of course it will blow up if I start using sub-directories, but oh well.
    • But with this fix in hand and a few more fixs to the Javascript test files for Thali I did get things running but really slowly. Something is very wrong between PouchDB and Couchbase Lite's listener. I strongly suspect it's TJWS but for now things worked and so I released an update to all the Thali files for Jon to play with!
Next Week
  • Get PouchDB's test suite running against Couchbase Lite to see what works and what doesn't.
  • Figure out what we are going to replace JavaFX with on the desktop
  • Release our replacement for JavaFX on the desktop - First Public Release
  • Put in QR Code Support along with an address book of some kind - Second Public Release
  • Put in TOR support - Third Public Release
May 20, 2014 at 1:52 AM
This Week
  • TL;DR - Supporting Jon's work on the demo for HTML 5 and filing/fixing a ton of bugs against Couchbase on Android and Java both from failed PouchDB tests and failing to run their own tests.
  • Monday - Figure out why Jon wasn't able to build Thali (he was using 'gradlew build' but needed 'gradlew distZip'), worked on getting PouchDB to work on Windows (got the build working but the tests won't run) and used Linux to tests PouchDB against Couchbase Lite using a project I created which result in 3 bugs against Couchbase just from the first 60 lines of code in the second test, there are 600 tests in PouchDB's test base. Also spent time running tests for PouchDB to get it working on Windows. Build now works on Windows but not PouchDB tests. So I run those from Linux.
  • Tuesday - Wrote a proposal on how to get rid of our Chrome extension dependency, set up the Chrome bridge so I could validate Jon's demo. Ran into bizarre problems with Virtualbox and KitKat that went away for no apparent reason. I ran into a nasty bug in the PPNET code, it wasn't running continuous replication!
  • Wednesday - Couchbase said my PR was breaking their tests against Sync Gateway which it was since I didn't test against Sync Gateway. I actually fixed the bugs but against Couchbase Listener not Sync Gateway because I wasn't sure how to get Sync Gateway to work. Lost a bunch of time in debugging tests that were failing because I was deleting stuff I shouldn't (again). Ran into a nasty bug in Couchbase that I filed.
    • But more importantly Jon and I had an amazing meeting with Ross Gardler who strongly suggested that the best thing we could do to make Thali successful is build a community and the best way to build a community is to get a total of 3 major contributors (we are 1) who are from three different independent sources. He said that in his Apache experience this was a good rule of thumb for a healthy community. So we are focusing on that.
  • Thursday - Wasted an impressive amount of time trying to set up Virtualbox VMs for Windows, the problem was (obviously) that HyperV was running. I didn't realize this because in the past Virtualbox wouldn't run at all if HyperV was running but now apparently it will (and yes, I do wonder if this has anything to do with Tuesday's issue). But got the VM set up and replicated Jon's demo in a clean environment. Found a bunch of bugs in Couchbase's Java code and fixed and/or filed those. Then Traun got me running on Sync Gateway which let me confirm all Android tests passed there! Finish the day by writing a script to automate building and packaging the files for the demo.
  • Friday - I cleaned up the Thali directory and checked it in. I then wrote an article on raw sockets as a result of an internal request. I then ran into another Java problem in Couchbase (they were translating /s in DB names into ':' which is illegal in Windows) and then I ran into a nasty attachment bug that has to do with Couchbase not properly releasing its file handles. For some reason Android doesn't care but Java sure does!
Next Week
  • Get our new dev up to speed on Thali! (Hey Stefan!)
  • Start working on TOR support.
  • Handle any emergencies that come up while Jon prepares for the HTML 5 conference.
May 28, 2014 at 6:07 PM
Edited May 28, 2014 at 6:07 PM
This Week
  • TL;DR- Worked on Tor spec and cleaning up how we handle tests/files so they suck less.
  • Monday - Lots of email. Weekly status. And long talks with Stefan (our new contributor!!!) and Jon.
  • Tuesday - Spent a bunch of time studying Tor and writing up the first drafts of the httpkeytor spec.
  • Wednesday - Internal Microsoft stuff. More Tor research. Put in place code to handle managing our dependencies files in Thali using a Zip file instead of a directory of Java resources. Getting the code to work took some wrestling with Android but worked. I wrote an article comparing Crosswalk and Cordova after Stefan pointed Crosswalk out to me. And somewhere in there I helped Jon out when our demo stopped working due to networking issues. That actually good a bit of time to get fixed up again.
    • The problem with the Java Resource is that in Android there isn't a good way to enumerate the files in a Java resource directory in a Jar. I already had a hack to create a manifest with the files that we could read but this would get unwieldy since Stefan needed lots of HTML files and directories. So the zip file approach let us create a single file with everything we could then unzip at run time.
  • Thursday - Realized that the Zip model was a bad idea. It would work but would always be strange. What we need is to use Resources in Java and assets in Android. We also need to clean up our test in general, the model we use for sharing tests is awful because it's error prone and heavily manual which discourages writing tests. So my solution is to have the tests live in Android and then use Gradle to copy them to Java and just abstract away the dependencies. This gets our test code out of production code and makes writing tests much easier. I spent a good chunk of today actually planning out how to do this right. Wrote up an article on what Thali needs from HTML 5. Did some experiments to see how well rear facing cameras on phones could handle QR Codes using Stefan's test code. And finally got back to cleaning up our testing/resources mess.
  • Friday - Worked on cleaning up assets and tests. Ran into an issue where I basically corrupted IntelliJ's project files (I moved some classes around outside of IntelliJ because it was too painful to do it in IntelliJ) and that took me a while to realize and then I ran into some bizarre constructor errors that turned out to be a swallowed exception because I was calling getActivity in a constructor (which I think I've done before =( ) and that's not good. I then ran into issues that there doesn't seem to be a mock activity to test against in Android so I had to put a single test activity into production. But I did get all the tests up and running in Android!
Next Week
  • Finish asset/tests clean up including getting tests running in remaining projects and removing unneeded code.
  • Finish spec on TOR support and start writing the code!
Jun 2, 2014 at 8:26 PM
This Week
  • TL;DR- Mostly researching how to support Tor
  • Monday - Mostly internal Microsoft stuff. Run a test for PouchDB to see if their latest fix for Windows works (alas, no).
  • Tuesday - IntelliJ went completely insane, finally figured out that I needed to close it down completely and restart. I did file the bug against Intellij. Made a bunch of progress on fixing up the test code from last week when I realized that thanks to our plans for a new design for clients fixing the remaining bugs was probably useless. So I stopped until we see how the new plan works out. Finished up the Tor httpkeytor spec and wrote a 'hello world' to the Tor mailing list.
  • Wednesday - Bunch of Tor research
  • Thursday - Spent a bunch of time researching I2P to respond to a mail from an I2P developer. Spent time experimenting with Tor hidden services to make sure they work the way I think they do. Spent time on IRC talking to Tor and Guardian folks to figure out more things.
  • Friday - Spent more time doing I2P research. Spent more time researching how best to support Tor. Spent time talking to Tor folks on IRC.
Next Week
  • Get Jon's PR request checked in
  • Figure out how to handle the issues we have with Guardian's Tor implementation on Android
  • Decide what to do about the TLS identity hole
  • See if we can actually get some of the Tor functionality up and running
Jun 10, 2014 at 4:33 PM
Edited Jun 10, 2014 at 5:04 PM
This Week
  • TL;DR - Got the client side of Tor support working in Java and Android and spent a ton of time on communications issues.
  • Monday - Death by a thousand small work items that had piled up like responding to Couchbase's comments on my PR, digging out of a bunch of emails, looking at Jon's PR to Thali, etc. I then started to go through Orbot's NetClient code to see how they talked to Thali and mapping out what changes this would require in Thali. This work helped me understand that httpkeytor was unnecessary.
  • Tuesday - Realizing that httpkeytor wasn't needed I completely re-edited the httpkey spec to do all the work we need. I then did some research and sent out some mails on the gory details of Tor hidden services since I think they help to solve one of our outstanding security issues (the SSL cert hole). I then (finally!) got to start coding. I set up a manual Tor Onion Proxy and started to write the code to talk to it.
  • Wednesday - Bunch of email, had a long chat with a potential new contributor to Thali and a long email with the Guardian folks on how they support Tor. I then started to put in SOCKs support to talk to the Tor Onion Proxy and got the basic structure started but then realized that even though Java does support SOCKS 4a for some reason DNS requests were not going through the SOCKS proxy!
  • Thursday - Wrote up a blog article on why I don't like digital signatures (this was in response to Google's announcement of a GPGMail extension to GMail) and helping Stefan navigate some issues in how to build Thali. I spent some time understanding more of Orbot's code and how exactly they handle the SOCKS 4a problem. I got some code kinda of working in Java but have run into some problems.
  • Friday - Needed to clean up our architecture page to make our conversations with other projects go better. I submitted a tiny PR for Couchbase at their request to support mavenLocal. Spend a bunch of time navigating the avalanche of bug cleanup notices coming out of Couchbase to see if any of them were an issue for us. I finally got back to coding and got Java working but the same code doesn't work in Android because Apache in Java supports layered socket factories and Android does not. After some investigation this turned out to be really easy to work around and now the code works in Java and Android!
Next Week
  • Fix up our tests to work with the changes I made to get the Tor client working and submit as a PR
  • Take the latest updates from Couchbase (especially a mac related one we need in Native)
  • Figure how we want to support the Tor Onion Proxy in Android
  • Start writing actual code to set up and run Tor Onion Proxies in Java and Android
Jun 16, 2014 at 11:48 PM
This Week
  • TL;DR - Meetings and email are killing me! Updated to latest Couchbase release and updated couchbase's tests to pass in Windows. Trying to figure out how to handle Tor's inherently flakey connection nature.
  • Monday - Spent a few hours writing a response to the Briar folks. Had a great (but long) talkie.io call with Jon Udell and Jeremie Miller of XMPP fame on how we can all play together. Spent a bunch of time trying to respond to issues that Jon brought up in his PPNet work before I realized I was jumping into a bottomless pit and stopped. But at least I have a bunch of good stubs for future blog posts. :)
  • Tuesday - Changed our existing tests to use Tor functionality. Turned up an issue that the tests are now intermittently failing. It seems to have to do with replication.
  • Wednesday - Gobs of meetings that ate up a bunch of time. To address the potential replication issue I updated all of our couchbase code to the latest bits which resulted in none of Couchbase's tests passing on Windows. I debugged this to a bunch of issues with couchbase and submitted a bunch of small PRs.
  • Thursday - Tracked down the remaining Couchbase bugs (they were all a variant of the same problem, Windows gets unhappy if you don't close streams) and now all couchbase tests pass in Windows! Still tracking down why my Tor tests are intermittently failing.
  • Friday - Had a long call with Jon Udell and Michael Rogers from the Briar project. The test failures in Tor all seem to be IO errors (no surprise). I changed Apache's retry logic but that didn't work. I looked at NetCipher's code to see how they handle this but don't see anything. I fixed some of our messed up logging so I could more clearly see what was going on.
Next Week
  • Dive deep to understand how to properly handle the Tor connection issues so our tests will pass reliably (and our code will work well in general)
  • Figure how to support the Tor Onion Proxy in Android
  • Start writing actual code to set up and run Tor Onion Proxies in Java and Android
Jun 25, 2014 at 12:33 AM
This Week
  • TL;DR- Finally checked in the Tor client code and started working on the Tor Onion Proxy code, went to Foo camp!
  • Monday - Ivan caused me to kill a good few hours writing up the Zooko's triangle article. I then turned on Tor logging and looked through TJWS's logs to figure out why we are seeing Tor related connection failures.
  • Tuesday - Spent time looking at Foo camp and sending emails to people with interesting session proposals. Did a 1.5 hour interview for a job candidate for our group. Spent the rest of the day talking to Jon, Stefan and Ivan.
  • Wednesday - Spent the entire morning on a great call with the PPNet folks regarding the changes we need. After turning up the retries on Apache client to silly levels and working through some odd test failures in couchbase's tests I submitted a few minor PRs to fix bugs but otherwise got the code working on the client side. Unfortunately doing a full test run across all our depos takes like 30 minutes. I did some baby perf testing to see if anything jumps out at me but it's not clear where the perf hit is coming from. I replicated all of Thali's repo's into our new Github Thali project! I then cloned Briar and am sniffing through their code for how they handle Tor.
  • Thursday - A ton of email hit and walking through Briar's code in preparation for creating our own stand alone Tor Onion Proxy Jar/AAR.
  • Friday - Got on plane and started Foo camp!
  • Saturday - At Foo camp!
  • Sunday - Foo camp finished up and I went home.
Next Week
  • I'm going to take Monday off to recover from the weekend
  • Start writing the code to host Tor Onion Proxies in Java and Android
Jul 7, 2014 at 7:40 PM
This Week
  • TL;DR- Got our Tor Onion Proxy code (that lets us host hidden services) working in Android and made good progress on adding Java/Windows support for it.
  • Monday - Recovering from Foo Camp
  • Tuesday - Catching up on email, finishing a bunch of minor work items that piled up, wrote up my Foo camp summary and started to explore the Briar code to figure out how to extract the Tor Onion Proxy (OP) code.
  • Wednesday - Took all day to extract the Tor OP code, get it refactored and building in Android.
  • Thursday - Drowned in email. Spent a bunch of time trying to figure out how to get tests running against the Android Tor OP code. I ran into annoying problems like old versions of configuration files that weren't being overwritten and issues with Tor hidden services taking a while to boot strap. But I finally got all the Android tests running and started to write the Java version.
  • Friday - Added back in the GeoIP functionality I accidentally removed to the Tor OP project. I then ran into an issue when I retried the Android tests where objects that were marked closeable failed on close. Apparently this is a known Android thing. Since I only needed these closes in test code I just disabled them and went back to work on the Java code.
Next Week
  • Write the code to get the Tor Onion Proxy hosted in Windows and maybe throw in Mac and Linux if it looks easy enough.
  • Get our heads wrapped around planning the rest of the summer for work on Thali and the Thali team!
Jul 14, 2014 at 7:29 PM
This Week
  • TL;DR- Got Tor Onion Proxy sorta working in Android and Java but there are reliability issues in the connections that require much smarter retry code.
  • Monday - Spent most of the day with Jim and Tim getting them up to speed on Thali.
  • Tuesday - Working on the Tor for Java code to figure out life cycle issues, the Tor folks on IRC pointed me to TAKEOWNERSHIP/__OwningControllerProcess. It turns out Briar is using part of this and after I asked they added the rest.
  • Wednesday - Two candidates I needed to interview and a pile of meetings with Ivan, Jim, James, etc.
  • Thursday - Wrote up a quick spec for the replication manager, did a full review on Stefan's push request and then integrated the latest changes from Briar and worked through a mess of minor bugs to get the same code controlling Tor on Android and Java.
  • Friday - Fixed a bug in my Java File Watcher for Tor code and another bug in finding the GeoIP file that finally got the Java code working but there are issues with reliability.
Next Week
  • Get us organized for demo work items
  • Figure out the retry issues with Tor, it fails, a lot.
Jul 21, 2014 at 6:17 PM
This Week
  • TL;DR - Our Tor Onion Proxy code now handles Android, Windows, Linux and Mac! We should do a public release of the Utility Library this week!!!!
  • Monday - Lots of calls with members of the team to get them unblocked. Wrote up the proposal for the demo for September and submitted the work items. I ran into reliability issues with Tor on Windows that I think are due to using a slightly older version of the Tor binary from the Tor Browser Bundle. When I switched to the expert release it had the latest bits and the problems went away.
  • Tuesday - Did a bunch of refactoring to clean up the Tor code. Switched to Ektorp work to get rid of our custom version of Ektorp and unblock Stefan. I then finished the refactoring and started writing the automatic retry code. Ended up spending time with Jason to help him with something he was blocked on.
  • Wednesday - Got a ton of features fixed! Even put in the infrastructure to support Mac and Linux.
  • Thursday - Put together a detailed proposal for what we need improved in Tor Hidden Services and sent out to Michael and the Briar folks for discussion. I was blocked on a nasty problem with restarted hidden services never being found. After spending an entire day on it I realized it was actually due to an idiotic bug on my part. :( But at least it's fixed and now things work really well, even with re-started hidden services.
  • Friday - Cleaned up some small bugs and tests are now passing on Windows and Android! Had a great call with Chris Anderson from Couchbase. Did more work to get things working on Mac and Linux. Over the weekend I managed to get the Mac and Linux code working.
Next Week
  • Publish the Tor library
  • Integrate the Tor library into Thali
  • Updated CouchBase bits
  • Figure out how I can help the team get done with our demo items.