I’m currently in the closing months of an extremely involved enterprise e-commerce project that is occupying all of my time (pretty much), which is why I appear to have fallen off the Internet… but I am still here.
One concrete milestone I have reached in the last couple of months is to have version two of my Yoke server project published. Version one added support for users to share via Android applications. Version two has added preliminary support for developers to promote their Android applications.
Early support for App promotion
The first glimpse of this functionality peeking through into my own applications can be seen in this customizable app promotion widget that now accompanies all of my mobile apps:
It’s generated by application data stored by Yoke and inserted via a simple JS include onto any webpage. Without JavaScript, the widget safely fails-over to a link that takes the visitor to a simple webpage that contains the same information:
It’s a convenient way to ‘socialize’ my application download links and manage them from a single console, which currently looks a bit like this:
What’s next
I’m now slogging my way through an email pile-up and trying to carve out some time for my own development work prior to returning to it full time. So what do I have planned?
Mainly to start yoking all of my applications together. Progress on my mobile apps is well under way and I’m working on rolling Moseycode and Betan in too.
I recently implemented OAuth for the Yoke service I’ve been developing. Yoke sign-in currently uses OAuth 1.0 for Twitter integration (via twitter4J) and a homespun OAuth 2.0 flow for Facebook and is used within my Android applications via a WebView. The regular Android approach would be to use the AccountManager API, but there are a number reasons I chose not to use it:
At the time, these all seemed like pretty compelling reasons to choose OAuth for my applications, and they still do. But now I think that it could be a risk for users to register via OAuth within mobile apps, especially Android apps. Why? It’s simply this:
The security of OAuth is predicated on the use of an honest browser client, within mobile apps this cannot be assumed.
Nothing prevents a malicious application developer from bundling, into their app, a browser component that mimics the behaviour of the native one. On Android, this is made easier because the source is open, and there isn’t any pre-vetting of apps published to the Market.
A compromised browser could readily skim off your password and log it to a server and it would be difficult, if not impossible, for you to discover that your credentials had been stolen in this way. The problem is that we are all conditioned to look past the browser, and to recognize the sign-in pages of the online services we use (Google, Facebook, Twitter etc.) and to give them our trust, unconsciously.
I will continue to provide authentication within Yoke apps using OAuth since I’ve invested time implementing it, but in the future I may be a little more cautious.
In my first technical post about Yoke and I thought I’d describe how I tackled building a scalable system for linking (yoking) users and items using App Engine.
Links between Users and Items are called Yokes, they are first class objects and are stored in the datastore as entities. This allows them to contain indexable information about how items relate to users.
In a SQL database this would be modelled naturally with two one-to-many relationships from Users to Yokes, and from Items to Yokes. The equivalent data model in App Engine involves adding two key properties to Yoke entities that associate each of them with a User and an Item. In App Engine there is the added consideration of entity groups. Entity groups are the unit of transactionality in App Engine and have a heavily constrained write rate as a result.
Since any given Item or User may be yoked extremely frequently, this has the consequence that Yoke entities must be root entities and cannot therefore partake in a transaction that also modifies the properties of an Item or User. But why would a datastore transaction which, say, creates a Yoke need to modify properties on the User or the Item or some other entity?
App Engine lacks the familiar SQL aggregation operations such as COUNT, SUM and AVG, so answering queries such as “how many times has this item been yoked?” requires that the aggregation be performed by the application by maintaining a counter in the datastore, and this necessarily involves at least one other entity. To also answer queries like “how many times has this user been yoked?” naturally requires two counters.
On what entities should we store our counters? Naively, we might think that the User and Item entities would be good choices, but this is not the case. To see why, it’s necessary to look at the distribution of yokes. Specifically, some items/users will be extremely popular and will be yoked very frequently. This is a problem if we store counters as part of the User/Item entities because those entities will become heavily contended.
To overcome this we need a new kind of entity, which the Yoke platform calls a Standing. Standings aggregate information about Yokes, for both Users and Items. So when the Yoke platform records a new Yoke it actually needs to create or update three different entities: the Yoke, the User’s Standing and the Item’s Standing. The process by which this occurs is necessarily quite complicated on App Engine because:
To help meet these requirements, Yoke transactionally enqueues a task when it creates (or deletes) a Yoke entity. The task wraps-up the work of updating the Standings. This means that the Yoke is created (or deleted) immediately, but the work of reconciling the Standings is deferred (usually only briefly), can take as long as necessary (up to 30mins), and will be automatically retried in the event of an error. These are standard characteristics of tasks in App Engine, unfortunately tasks also introduce a couple more complications:
To handle this, each task that adjusts a Standing is allocated an ID. At each step in the process of adjusting the Standing (including the first step which involves adding or deleting the Yoke) a temporary Marker is created containing the same ID. Before the task starts its work, it attempts to get each possible Marker from the datastore and, based on which Markers are present, it resumes its work at a particular step. After the task has finished its work, all the Marker entities are deleted.
Since it’s a possibility (though only a remote one) that a task may fail to clean up all of its Markers, a cron job sweeps through periodically and deletes any Markers it identifies as being stale. The only loose-end this leaves is the possibility that two tasks will ‘race’ to create the same Standing entity (creation of Standings is deferred since many Items and Users will never be yoked). This is handled by a separate mechanism that I may describe in a later post.
The absence of unrestricted transactions in the App Engine datastore does significantly increase the work needed to perform operations that one would consider to be basic in SQL databases or their equivalents, but any given approach (there is more than possible approach here) can be generalized and reused. So perhaps its best to think of App Engine as providing a lower-level persistence service that can then be tailored to meet specific needs.
I finally found a little time to do some Android coding. The result is a new sign-in flow for Yoke:
Here I’m trying to make it as simple as possible for users to establish an identity with an application. The activity is part of a generic Android library I’ve developed that matches an app’s look using style declarations to override the default appearance.
There’s also a nice technique in use here: The explanatory text is in a ScrollView. To bottom align it when the text doesn’t fill the view I combine the attribute android:fillViewport with a vertical LinearLayout (to hold the text) that contains a trivial zero-height View of weight 1. The result is that, when the text is smaller than the ScrollView, the trivial View soaks-up all of the empty space, pushing the text downwards, and when the text is larger than the ScrollView, the trivial View get squeezed to nothing.
The Google favicon is a placeholder — does anyone know of an approved icon I can use?
I’ve been working on a project called Yoke. It’s a server application for enhancing mobile applications and it’s accompanied by an Android library that takes most of the effort out of using Yoke in Android applications.
I started working on Yoke because there were a number of features that I really wanted to add to my Android applications for which I lacked a good approach. My key problem was:
How to make user generated content shareable?
Mobile devices typically have spotty internet connections, so if you want to make the user generated content accessible in a reliable way it needs to be copied to a server. And unless you want to prevent users from editing their content after they have shared it, you need a way to update content that’s already been uploaded. And since it would be quite annoying for a user to request these updates manually, it needs to be done automatically on their behalf (without failing “annoyingly” if the network fails). And since users might use the application on more than one device it’s necessary to synchronize the user content between all devices and the server, in both directions. Using the application across multiple devices in this way further implies that each user needs some form of identity within the application.
Then there’s the actual user mechanics of sharing to consider. Although sharing within an application can be as simple as making content visible to all the other users, the reality is that, for even mildly popular applications, the volume of content generated requires indexing and sorting to make it navigable. And because user generated content naturally varies in quality you need a mechanism for surfacing the best content. Approaching this (as I have) by allowing users to ‘like’ or ‘favorite’ the content of other users involves many other considerations: how to cope with flurries of ‘likes’ on extremely popular content, how to provide a stable ordering for browsing by popularity when it is constantly fluctuating, how to accommodate content that isn’t public, etc.
Simple sounding problems can hide a lot of complexity. Yoke is my attempt to write a reusable platform that does only as much as I need to let users share content via my mobile applications. I just want to give them an easy way to:
in a way that doesn’t constrain the applications I can write.
I think I’m almost there now, Metaglow has been my testbed which I’m looking forward to releasing soon, after that I’m going to try and find time to write a few posts about what I learned by implementing Yoke.
