Parentheticals & Excursions

How I quickly evolved an initial concept of “an eye...

Sun, 20 May 2012 09:11:48 +0100

How I quickly evolved an initial concept of “an eye peaking through angled brackets” into a an icon for my small Web Intents Snooper chrome app.

I’m not a designer, but the tools that we all have available to us through our computers are simply incredible. So why not have a go?

My first impression of the Chrome Web Store

Mon, 30 Apr 2012 22:50:23 +0100

I’ve been interested for a long time to find out how the chrome web store stacks up against Play (aka Android market) for app developers. So, largely by way of an experiment, I published my harmonica tuning tool on there today and I thought I’d capture some rambling observations in a blog post.

In what follows:

CWSDD = Chrome Web Store Developer Dashboard

PADC = Play Android Developer Console

(an aside: I’m sure it’s being worked on, but Google really needs to start unwinding some of the complexity that’s creeping into the naming of their developer services)

Packaging

I wanted to start off with a small comment about packaging your web app for inclusion on the chrome web store: it’s incredibly simple. JSON+PNG->ZIP and that’s it. I’ve written a couple of chrome extensions for internal team use - that was super simple. This is even simpler.

The CWSDD uses the Google Webmaster site ownership verification tool to safeguard against people publishing other developer’s applications under their own apps. The only minor thing I tripped-up over was that using *:// requires verification of both http:// and https://.

Icon

This is a very specific tool that I doubt will garner even a small handful of users, so I’ll be honest: I dashed off an icon and the minimum number of promotional images I could get away with. Even then I struggled with the icon. The guidelines specify no perspective. Is there any way to draw a harmonica without perspective that doesn’t make it look like an abstract composition of shapes? Well, I politely sidestepped that guidance. It also felt odd producing just a single icon asset; I’ve become accustomed to creating multiple assets for Android app icons and have come to see it as as a feature.

Promotion

The documentation for the CWSDD does an excellent job of communicating which assets you need to provide and why, and how you can expect they’ll be used. I don’t recall anything as clear as that for the PADC, but things may have improved considerably since I last read the docs. Of course the CWS has the advantage of not being required to support legacy clients over a wide number of form-factors, as the Android Play developers must.

There were no onerous requirements. I got away with supplying just two images, a small promotional image and a screenshot (this little app isn’t worth more).

Publishing

In general, the process of uploading and publishing was very similar to that of publishing an Android app; both in terms of how the forms are presented and the flow that ties them together. On the whole I’d say that the CWSDD feels more ‘coherent’, but that the PADC clearly packs in more features.

Around publishing, the CWSDD shows up a couple of very basic deficiencies in the PADC. The first is that the CWSDD supports “trusted testers”. It appears to allow you to define your testers by membership of any Google Group you set up. I didn’t use it (this is a bit embarrassing, but I don’t actually know anyone else who’d want to use this app) so I don’t know how well it works in practice, but it looks like a very clever solution.

Secondly, CWSDD allows you to supply a Google Analytics ID to track user views of app listing (Google Code supports this too). It’s a real pity that PADC doesn’t do something similar; this is valuable information that Android developers are missing out on.

Finally

These are just my first impressions of the chrome web store developer dashboard, with an application that I think has a place in the store, but which is nevertheless very a modest app. I found it simple, well documented, and quite slick.

I will definitely design any future web apps with an eye to integrating them into the chrome web store and will also suggest it to relevant clients as a strong option (with the caveat that I don’t know how smoothly non-free apps operate).

Experimenting with Harmonica Tunings

Fri, 27 Apr 2012 21:35:09 +0100

I’ve just published a small online tool investigate different harmonica tunings.

http://www.tomgibara.com/harmonica/tunings/

As indicated by my previous two posts, I’ve recently become interested in learning about harmonica tunings. So I wrote a small tool that allowed me to analyze the chords that are available in a given tuning. Then I thought I’d share it, so I extended it with some standard tunings and gave it just a little polish.

Note that I’ve been hasty and haven’t taken any time to test it on different os/browsers/screen combinations, if anyone wants to report an issue, a comment on this post would be a good way to report it.

Why an harmonica tuning?

Fri, 20 Apr 2012 00:45:25 +0100

I’ve just finished designing a new harmonica tuning.

It’s difficult to say why. I think I just really enjoy finding elegant and balanced solutions to problems. A harmonica tuning provides a really excellent challenge - even if you’re mostly musically illiterate like me.

Firstly there are a number of rigid constraints: there are fixed number of holes and physical laws determine which combinations of notes they can provide.

Then you have to consider the perspective of someone playing the harmonica. What patterns of notes are easy to learn? and easy to hit? and easy to express with?

And then there’s the music that you’re trying to make it possible to play - it’s accumulated over centuries and it’s varied and untrammelled. You can’t fit much of it into the 10 holes of a harmonica, so some hard choices need to be made.

Now that I reflect on it, it’s very much like producing software: there’s the hard rules programming, the needs of the user to consider, and behind it all, a universe of things the user wants to achieve with your software, if you can just get it right.

I love developing software, and maybe that’s why I’ve enjoyed designing a harmonica tuning.

I recently started learning the harmonica. I’m rubbish at...

Fri, 20 Apr 2012 00:13:58 +0100

I recently started learning the harmonica. I’m rubbish at it, but am enjoying it hugely.

Mainly for the pleasure of it, I’ve taken a stab at designing my own 10 hole tuning, and I’m delighted with how well it turned out.

The tuning (shown above for C) spans very nearly three octaves and I guess it would be broadly classed as a spiral tuning.

The blow and draw notes on all holes are separated by a two semitones. This allows every draw note to be bent, but avoids any excessive difficultly that comes from having a larger interval (I’m thinking here of the 3 hole bends in a Richter tuning).

Central to this tuning is that every seventh note in the full chromatic scale is relegated to an overblow (consequently full chromaticity requires 5 overblows). This provides a rich and useful set of chords. Indeed, drawing or blowing on any three adjacent holes always yields a major or minor chord (which consistently alternate along the harmonica).

The tonic is available in the first octave on draw 1-3 (and it’s also available on blow 5-7). The dominant is adjacent on draw 3-5, with the subdominant available both major (draw 5-7) and minor (draw 2-4). I think that should make the tuning well suited to regular blues and rock.

I haven’t weighed up the negatives yet, two things are evident: there are no opportunities to play octaves, and the note pattern (though systematic) is different in all three octaves - so it’s business as usual there.

Binary digits

Wed, 18 Apr 2012 21:55:09 +0100

I just realized that when I count on my fingers, I count in binary.

How often as adults do we count on our fingers? Today, for the first time I can remember, I did. It was a reflex, and when I looked down at my fingers, I found myself (to my own surprise!) counting in binary patterns on my fingers.

My mind was jolted back to a dim memory: that in my teens, I made a concious decision that I would count binary on my fingers. The advantage was as obvious to me then as it is now. I can count to 1024 without pausing for thought.

I hope it didn’t take me long to learn since, as it turns out, I clearly haven’t needed the skill.

Graphics utility code

Sun, 15 Apr 2012 23:54:00 +0100

I just committed some rough source code for, amongst other things, ImageUtil.

You’ll find it in my Google code SVN repository

You need this code to create the colour charts I posted a couple of weeks ago.

It’s inside a Maven project, but you can probably just cherry pick this file and have things work:

http://tomgibara.googlecode.com/svn/trunk/graphics/graphics-util/src/main/java/com/tomgibara/graphics/util/ImageUtil.java

And watch out for this possible issue:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4886732

I haven’t been able to track down all of the situations it occurs in.

Crinch 0.7.2 Released

Thu, 12 Apr 2012 15:50:30 +0100

Crinch has had another minor bump to version 0.7.2. This release fixes an issue that can prevent some character sequences being returned from tries.

Some computer generated eyes. Based this early work.

Wed, 11 Apr 2012 01:02:55 +0100

Some computer generated eyes. Based this early work.

Thankfully, I’m better at coding than drawing, otherwise...

Sat, 07 Apr 2012 22:54:49 +0100

Thankfully, I’m better at coding than drawing, otherwise there would be no hope.

I made some time to try and apply what I’ve discovered...

Sat, 07 Apr 2012 00:24:56 +0100

I made some time to try and apply what I’ve discovered about irises to generating them using my genetics library. These are my very first results.

Only the stupid

Sun, 01 Apr 2012 11:18:00 +0100

I’m writing this based on very limited information, I only know what the BBC has reported here. But I feel that I don’t need to know more than this:

A new law - which may be announced in the forthcoming Queen’s Speech in May - would not allow GCHQ to access the content of emails, calls or messages without a warrant.

But it would enable intelligence officers to identify who an individual or group is in contact with, how often and for how long.

One can only assume, based on the repeated proposals for legislation such as this, that the government are only prohibited from installing CCTV cameras in every room of every home in the UK on the basis of cost alone.

MPs who support legislation such as this must be simple minded, ignorant, or in the sway of people other than their constituents. There’s no reason to believe that once such a law is enacted that it won’t persist indefinitely, or even be extended. Can the present government guarantee that there will never be political party elected to power that won’t see advantage in abusing such breathtaking powers?

McCarthyism was only 60 years ago. They are trading convenience now, for the almost certain use of the law to harass and persecute others in the future.

Stupid.

Some quick notes I sketched concerning iris generation. This is...

Sat, 31 Mar 2012 01:03:00 +0100

Some quick notes I sketched concerning iris generation. This is obviously related to yesterday’s post about eye colour.

Crinch 0.7.1 Released

Fri, 30 Mar 2012 07:49:37 +0100

A quick post to say that the Crinch libraries have had a minor bump from 0.7 to 0.7.1.

It was motivated by an urgent fix for a bug that impacted one of my clients. The bug was in the incomplete and undocumented record library, so it shouldn’t have any impact for other users of the library at the moment.

The only new aspect to this release is that the source code for each library is now being put into jars and published together with a new distribution jar that aggregates all of the Crinch libraries. This jar is available for download from Google Code.

Eye Colour Variation

Wed, 28 Mar 2012 23:55:00 +0100

For reasons I won’t elaborate on now, I want to find out how eye color can be parameterized. After some fruitless online searches, I resorted to discovering it myself.

I’m not looking perform a scientific study, so my steps are simple: sample some eye colours, analyze them with some plots, create a model and then validate it. The first two steps haven’t taken very long at all.

First I used GIMP to sample average colours from 44 photographs of irises and collected them in a small swatch.

Then I wrote a small Java app to plot the data. Here are the plots:

The plots demonstrate that the variation of basic eye colour is actually fairly narrowly defined. This doesn’t actually surprise me, but it’s certainly interesting.

Of course this is a small sample, but it’s good enough to establish a basic approach. I think I’m going to model eye colour in three separately parameterized families: blue, brown and green (no surprise there) using constraints on hue, saturation and brightness based on the plots.

And just in case anyone’s interested, this is the small throw-away app I wrote to create the plots. Note that it depends on a class from a small Java graphics library

Crinch 0.7 Released

Sun, 25 Mar 2012 19:48:41 +0100

Another weekend and another code release. I’ve released a new version of my crinch libraries. It now includes a documented API for binary coding. It currently supports a range of universal and non-universal codings. All I/O for the package is directed via the Crinch bits library which makes the library very flexible.

Universal codes

The library currently supports the following universal codings.

All universal codings support arbitrarily large numbers via BigInteger.

Non-universal codes

In addition to universal codings, the following non-universal codings are supported.

The Huffman implementation in particular has seen lots of work: it accelerates encoding through canonicalization and block reads bits for faster decoding (without a some degree of read-ahead, I’m not sure there’s a faster approach). It can also return a dictionary which contains the minimal state needed to reconstruct the encoding. This can be efficiently transmitted ahead of a compressed message.

Other features

A number of helpful classes are provided in addition to the core coding implementations.

ExtendedCoding wraps a coding to provide additional
CodedReader / CodedWriter conveniently pairs an ExtendedCoding with a BitReader/BitWriter
CodedStreams provides static utility methods for common encoding and decoding tasks.
CharFrequencyRecorder accumulates character frequencies from Strings and other sources of character data; useful for Huffman coding.
CodingFrequencies calculates ‘zero-order’ information entropy from data arrays.

Future extensions

The API is mostly finalized, though there are some functions which are likely to be added:

Chaining classes to allow coded readers and writers to be used with greater fluency eg. writer.chain().encodeInt(x).encodeLong(y).flush().
Implementation of more codes including possibly: range/arithmetic, Elias gamma, exp-Golomb, Levenstein, Yamamoto

If anyone has an coding they want/need implemented and that isn’t currently supported, you can let me know with a comment.

BitWriter performance

Fri, 16 Mar 2012 01:23:19 +0000

Crinch provides a number of different BitWriter implementations that can be used to accumulate written bits in memory. They each have advantages in different situations:

ByteArrayBitWriter is fairly fast, but can’t be sized at a bit level.
OutputStreamBitWriter can be used to write unbounded amounts of data, but is slower.
IntArrayBitWriter has a fairly balanced performance profile.
BitVector.openWriter is flexible and faster for writes spanning many bits, but this comes with some overhead per write.

I ran a few benchmarks to verify the performance of these classes for (a) writing individual bits (b) writing multiple bits and (c) writing padding bits. I included the NullBitWriter to provide a baseline which gives some indication of the intrinsic API overhead.

As one might expect, BitWriter implementations that operate on greater numbers of bits simultaneously perform slightly worse when only single bits a written. A single ternary operator (writeBit(bit ? 1 : 0) vs write(bit, 1)) accounts for the significant difference in performance between WriteBoolean and WriteBit.

Although the ByteArrayBitWriter again performs well, observe how the performance of writing to BitVector improves significantly as bits are written in larger groups

The poor performance of IntArrayBitWriter for these two operations has highlighted a poorly optimized implementation. BitVector maintains a ‘flat’ performance profile; it provides the only writer that pads 255 bit intervals as quickly as word aligned 256 intervals.

All I want to do is tweak a method

Wed, 14 Mar 2012 21:19:55 +0000

My encodePositiveInt method is now going to allow calls with zero. So how should I change the method name to reflect this?

encodeNonNegativeInt long
encodeNonNegative overloaded
encodeNonNegInt ugly
encodeUnsignedInt misleading
encodePositiveInt inaccurate

Goddamit. What do I call this method?

The impact of Views on BitVector performance

Mon, 12 Mar 2012 01:41:49 +0000

Following on from my post about the design of the BitVector class yesterday, the question was asked, do you really need operations over ranges when you could just use views?

For example, the following two lines of code will have exactly the same result:

v.setRange(100, 200, false);
v.rangeView(100,200).set(false);

Except that the second line will create an intermediate object. Putting aside the fact that not every environment that executes Java code has performance to match the standard Java runtime, the question of whether intermediate views should replace ranges hinges on the degree to which they impair performance.

Comparing Performance

To ascertain this, I put together a hasty benchmark this evening. It repeats a large number of different operations over a BitVector, using both ranges and views. For stable results, the fastest and slowest third of each run was discarded, and the remainder averaged for each operation.

The tests were performed on a 32 bit dual core machine running

OpenJDK Runtime Environment (IcedTea6 1.11pre) (6b23~pre11-0ubuntu1.11.10.1)
OpenJDK Server VM (build 20.0-b11, mixed mode)

The following chart gives a summary of the results:

It shows there is a clear and consistent cost to the object allocations (about 100ms in each of these tests). Given the performance sensitive nature of bit handling, this isn’t something I’d necessarily want to give away for the benefit of fewer methods.

Rotate Performance

In addition to these operations, I also measured the performance of the much slower rotate operation. This threw up an interesting anomaly: in contrast to all other operations, creating the intermediate view is always faster!

From the perspective of a developer who doesn’t have visibility of the algorithms applied by the JIT, this is inexplicable. Here is the rotateRange method which is being called:

public void rotateRange(int from, int to, int distance) {
    if (from < 0) throw new IllegalArgumentException();
    if (from > to) throw new IllegalArgumentException();
    from += start;
    to += start;
    if (to > finish) throw new IllegalArgumentException();
    rotateAdj(from, to, distance);
}

In the other case we are calling rangeView() followed by rotate(). Here is one subcall of the rangeView method:

public BitVector duplicate(int from, int to, boolean copy, boolean mutable) {
    if (mutable && !copy && !this.mutable) throw new IllegalStateException();
    if (from < 0) throw new IllegalArgumentException();
    if (to < from) throw new IllegalArgumentException();
    from += start;
    to += start;
    if (to > finish) throw new IllegalArgumentException();
    return duplicateAdj(from, to, copy, mutable);
}

Notice the basic similarity in structure. Both methods are doing the same checks except that the rangeView method actually ends up doing more because it then needs to create a new BitVector. Finally, here’s the rotate method that will be called on the newly created view:

public void rotate(int distance) {
    rotateAdj(start, finish, distance);
}

These code snippets show that the the rotateRange method cannot do more work than a call to rangeView and rotate combined. So how can it be consistently slower?

Closing Comments

I don’t even have the start of explanation for this anomaly (except that it may be a bug or shortcoming in the JIT), but as I observed in my comments yesterday, JITs can be capricious. Writing APIs that anticipate good results from them is fine, but there’s no escaping the fact that it actually passes risk onto the developers who use the library. The size of that risk is a function of performance: the degree to which JIT optimizations influence it and its importance to the developer.

Why does BitVector have so many methods?

Sat, 10 Mar 2012 23:55:05 +0000

This is a post about the API design of BitVector. I don’t think I’ve ever written a class which carried so many methods and here I try to give some justifications.

Binary data, in the form of an order sequence of bits, is an extremely fundamental software abstraction. Perhaps as a consequence of this there’s frequently a lot of it, and there are often many operations that need to be perform on it.

The large number of useful operations possible over binary data is one reason that BitVector has many methods, but there are others.

Developers spend most of their time processing binary data as byte sequences (generally byte arrays in most languages). Grouping bits into bytes, and in more generally into longer words (such as ints and longs), has the significant benefit of improving performance. An operation on a subsequence of 64 bits could be done with a single operation on a long. Addresses are shorter too; I can’t imagine any developer hankering for bit level memory addresses. And when a given word length neatly accommodates the data, everything is much simpler.

But some situations arise where fixed word lengths just don’t fit. Universal codes are one example: bit-length can vary dramatically from one value to the next, and though lengths are frequently less than a single byte long, they are unbounded. In this case, and in others, there isn’t an easy way to avoid bit-level processing.

So we’re still in the situation where we might have lots of data, and lots of things to do with it, but now have the added pressure of reduced performance. Take the example of a simple loop over a byte array which sets every byte to zero. The same loop at bit level will take eight times longer. What is more, the language (Java in this case) isn’t going to give you any core primitives to help: exposing direct access to byte arrays can boost efficiency, but bit-level operations are necessarily going to be forced through a method call. And, even more significantly for performance, the work of setting a single bit will also require significantly more code than the simple assignment needed for a language primitive like byte.

This pressure on performance significantly changes the trade-offs made when designing a general purpose API for bit-level processing. Whereas, the value of providing a method that, say, performs an element-wise XOR between two byte arrays is questionable (given the simplicity of implementing such a method), the value of the same method at bit level is much higher because very significant optimizations become available.

So performance is another reason that BitVector has so many methods. Simplistically: more specialized methods allow for more optimizations(*). But there is another reason there are so many methods.

Prior to starting development of the bit package, and BitVector in particular. I reviewed my various code bases to see what I needed from such a package. What I found was that my code (which often exposed bit level data in thinly wrapped byte arrays) was frequently forced to make array copies solely to guarantee encapsulation. And additionally, bytes were being copied into other intermediate forms, and then copied back just so that certain operations could be performed by methods that didn’t support the originating data representation.

I realized that, by creating a single class that supported encapsulation and provided these disparate methods, I could eliminate a great deal of the copying in my applications. And when there’s a lot of data, this gives a very large gain in performance.

Now, as one would expect when designing a class which is becoming overloaded with methods, I investigated ways of splitting the functionality into separate classes, and superficially this appears easy - the functionality of BitVector can be split along a number of clear lines and an API looking something like this is obvious:

v.asBits().setRange(0, 5, true);
v.asNumber().intValue();
v.asSequence().firstZero();

But a difficulty arises when BitVectors are aggregated in large numbers. For example, I had one small application that did lots of processing with 128 bit numbers. It stored a great many of these values, (representing them as a class with two long fields). For BitVector to be usable in this context, and others like it, per-instance memory usage becomes a consideration: Every ‘sub-view’ of a BitVector would require one of: storage of a reference to the view in a field and references back OR creating a new view on each call OR something more complicated.

I won’t go into detail, but however you tackle these options, nothing very good comes out of them: memory swells, or uneven performance occurs in ways that a developer wouldn’t anticipate. So I went for the low-tech approach - clear and logical method names together with documentation (still ongoing).

I’ll have to wait and see how it turns out.

(*) I also chose to provide more generalized methods too - this decision is more questionable - and was made on the basis that there is utility in the generalized form for some applications.

Parentheticals & Excursions

How I quickly evolved an initial concept of “an eye...

My first impression of the Chrome Web Store

Packaging

Icon

Promotion

Categories

Publishing

Finally

Experimenting with Harmonica Tunings

Why an harmonica tuning?

I recently started learning the harmonica. I’m rubbish at...

Binary digits

Graphics utility code

Crinch 0.7.2 Released

Some computer generated eyes. Based this early work.

Thankfully, I’m better at coding than drawing, otherwise...

I made some time to try and apply what I’ve discovered...

Only the stupid

Some quick notes I sketched concerning iris generation. This is...

Crinch 0.7.1 Released

Eye Colour Variation

Crinch 0.7 Released

Universal codes

Non-universal codes

Other features

Future extensions

BitWriter performance

All I want to do is tweak a method

The impact of Views on BitVector performance

Comparing Performance

Rotate Performance

Closing Comments

Why does BitVector have so many methods?