What you do matters far more than what you say.

After being hosted on the end of my ADSL line for the last few months, TheGovernmentSays.com has moved to a large shiny new server as part of the “mysociety friends and family” projects.

Various other projects will land there shortly, but it does mean that TGS now has reliable hosting. Apologies to those who have been affected by the problems caused by me moving house and the fan falling off the cpu of the box that previously hosted it.

One of the benefits of this is that we can much more easily scale up the processing. A single fetch/process/update cycle for TGS used to be about 30 minutes. And since we rerun the process every hour early in the mornings, that was about the limit.

You may have noticed from the vast numbers of posts today that I’ve added in all of the missing major departments (about 30 new feeds were added today). I have another 40 or 50 to add when the next major feature arrives – which is versions of TGS for each of the regions (and Wales, Scotland and Northern Ireland). Various bits of stuff will be added over the next few weeks and months, but hopefully it’ll be all done by the end of the summer.

William Heath recently talked about what would happen if Google killed Directionlessgov (here).
While I doubt they would – we don’t make money, just a mockery. That said, there are threats to Directionless; ones which are far more deadly than Google could be.

The part of Google we use is their search engine, and when I talk about Google here, that’s the only bit I mean – www.google.com/search . Search is big business – Google (GOOG) alone is worth about $112 billion. Even so, if Google suddenly didn’t exist for any reason, we would just use a different search engine. It may suck more, but we’d be OK. The reason for them existing no longer may be that they don’t want us to use their site, maybe they make changes which are incompatible with what we do, or
maybe the ISS crashed into them. Search is a commodity. As with many of the sites on the internet, if you can’t use them for purpose, whether they exist is irrelevant.

Much of what Directionless, and other sites such as http://www.TheyWorkForYou.com, http://www.PublicWhip.org, and overseas equivalents such as http://www.GovTrack.US, do, is scraping. We read the html page that Parliament (or wherever) serves out for browsers, and interpret it and pull out the content using automated programs. Those programs are, and have to be, quite conservative in what they understand – as accurate representation is critical. When those page structures change, our sites break until we catch up. While change is annoying, progress is a good thing (“progress” backwards, however, is just annoying).

There is no free, reliable, source of the postcode to Local Authority (or Constituency) lookup table – it costs a very large amount of money. With a budget of 3 tea bags, some milk and a chocolate biscuit, Local Directionless needs to know which Authority covers your postcode so that we can search the relevant website. We scrape this information from another site. However, when the site we scrape changes, stops working or moves block us, then we stop working until we fix it. While we can adapt, and are be on the defensive in an arms war and will only win because of more motivation and supplies of tea.

The biggest threat to directionless, and all similar civic sites, is not that one commodity part stops working until we switch (Google to Yahoo for example), it’s that one vital part prevents us from working. For some projects, it’s mapping data (unavailable, rather than restricted), for some, it’s the license of Hansard and potential problems caused by legal threats, in others, it’s what happens when a partially open site takes offline part of the site we use, and where we need up to date lookups.

What took most of the time of the local search was not the implementation, but creating the lookup table between Local Authorities/Councils (and how they were titled: “Cambridge” is not the same as “Cambridge City” or “Cambridge County”). That’s the real core of Directionless Local. And there’s no easy way for us to get that list from any other public site.

That lookup isn’t commoditised. Yet. We could protect it and commoditise it by creating a list of postcodes, one post code per authority and constituency (and possibly even ward?), and make it publicly available. There are many websites which allow you to do a lookup, and definitive lookup tables could be created, by anyone, from any of them. But only if there is a master list which we know is comprehensive. Then Directionless, and any other site, could swap in any site we wanted to get the information from. Some of these jobs aren’t sexy, but need doing.

Why does the local search currently not work in places? Because the site we scrape (which we were able to get a definitive list of authority names from) has taken down some authorities while they correct the data they show. Unfortunately, until they do, we stay broken in those areas. And there’s not a thing we can do about it, and we’re stuck until they fix it.

That’s the biggest threat to www.Directionlessgov.com – well meaning, justifiable, simple, and deadly.

Not really anythign to do with democracy, but more to do with actually being useful for me, I’ve created some RSS feeds for commits to the openbsd stable branches. For details, see http://flirble.disruptiveproactivity.com/rss/

Much has recently been made of
direct.gov.uk’s local searching
capabilities. Putting you straight to the page you want (via a handful
of interstitial pages).

In the best directionlessgov.com traditions, here’s something better:
http://www.directionlessgov.com/local/.

Direct.gov.uk requires you to pick the category that your thing fits
into (most) from their list, and then follow through several poges
click through to what you’re actually looking for.

previously, we’ve done a direct comparison of google and
Direct.gov.uk, but, given the extremely strict categorisation of
direct.gov.uk (and the extreme lack of strictness or catgeories in
what people actually look for), it’s not worth the
many days it would take (against the day or so it’s taken so
for). So you get national direct.gov.uk results, and local
google results.

We map your postcode onto your local authority, and then use this list of councils and websites (which took a couple of days to
create forthis purpose) to search just your council(s).

So, there you have it Directionlessgov.com/local.

[Note: most of this was originally written in summer 2005, with some minor tweaks and published in February 2006]

Work done by Government is generally one of 3 things; doing
work people don’t see, doing work people see, or moving
something from one category to the other. Search engines and
portal tools are often used to help citizens do what they
can’t find. Vast quantities of money, goodwill and time are
wasted because of poor re-implementations of a moderately
hard problem.

In a direct comparison early in 2005, 75% of users, when
shown results of search queries side by side, selected a
result from Google (limited to .gov.uk domains) over the
specialised £4.4m portal direct.gov.uk[2].

Users submitted a term to a webpage, and were shown, side by
side, the results of their search in the direct.gov.uk
engine and in google.

When the term was submitted, a request was sent to the
direct.gov.uk search engine, and the resulting HTML parsed
and the results extracted and displayed. Google’s knowledge
of .gov.uk sites was then searched[1] and similarly displayed.

The results were shown in the order that the search engine returned them,
and the default number of results for each engine shown side
by side (20 results for direct.gov.uk and 10 for Google). When a
user clicked on a page they wanted to look at, they
were first taken to a script which recorded their search term,
preferred url and which search engine returned that link, before bouncing
them to where they wanted to go. Users who opened multiple links for the
same search term (for whatever reason) generated multiple records.

Initially, we displayed google results in the left, and
Direct.gov.uk results on the right. After a month of usage,
these were switched to look into whether the earlier display
of google (on the left) was disadvantaging direct.gov.uk.
When the direct.gov.uk results were shown first, 28% of
people a small (3%) decrease in the percentage who clicked a
google result when it was shown second rather than first.

People generally searched using “keywords”, rather than a
sentance or phrase. Where longer search terms were submitted
(ie four words or more), these were generally for very
specific searches (e.g. “national spatial address
infrastructure”, or “freedom of information request
exemption”) , rather than english phrasing around what was
wanted (e.g. “how much council tax is band e in selsey”).

Direct.gov.uk “improved” searching

In the summer of 2005, partially as a result of criticism,
direct.gov.uk loudly announced a new, improved search engine
in August. Taking only results from September 2005 onwards,
repeating the above comparison, the newer engine made
no difference – the direct.gov.uk was still selected 28% of
the time (we still showed direct.gov.uk results first).

Footnote

1. used the Google API which returns identical results to www.google.com but in a more computer readable format

2. These stats were generated in a 3 month period in 2005.

Link to page about the event

Jim Griffin

personalisation gives much more choice, but fragments community. If we
don’t like the news that is offered by one station, we switch to another
which more reflects our views. This may not be the best
thing for the community at large,

Martin Higginson

1.6 billion phones vs 16 million ipods. People want on device.

Music will change.

lots of videos – little discussion or content

Joe trippi

Mobile comms offers a shift in power of goverence.

Information wants to be used

If you can get info when and where you need it at any time, power moves
towards the bottom – down from the people who have traditionally held it

No structure is immune – music is the foreteller as it is the most popular
and relatively small, connectable, contained of info (songs).

Journalism is already affected – camera phone photojournalists (SS: e.g.
snoopt.com in this month’s Wired)

This is a good thing for building communities and empowerment. Different to
one way communication media which focuses a single community.

Wireless media currently at same stage as TV in early 50s US. Nixon TV
speech to American people – he talked about the puppy, “Just like that,
Bullshit had it’s own medium”.

Tech changes lead to let us talk back and talk to each other. People want
to make things better. With that power comes responsibility and
accountability.

Global warming ringtone – have a tune which says “I care about global
warming”. How many ringtonea do you hear in a day which mean nothing. (SS:
What about this idea for Children In Need or similar other broadcast event?)

Discussion after talks

432 people were supporting Dean – they are different to the rest of the
world. If everyone asks one additilonal person got another person, they
affected the campaign…. Trippi gives an example with the
Signs for Dean story.

How are we exposed to new stuff? Sites, music, etc. What happens when there
is no shared understanding of concepts, when not everyone saw the same news
programme and can talk about it? (Griffin)

Even if 80% don’t agreed with a minority opinion (e.g. Iraq war) it’s
impprtant that that opinion can be beard. What happens if there’s no
replacement for John Peel? (example added by me to clarify)

Do we need X,000 Joe Trippi’s to make sure this goes right?

If a friend recommends something, then you listen. Whatever the thing being
recommended is.

What happens when tech is used for bad purposes? No tech is unique in this
respect.
Getting checks and balances right is not easy

Disinformation can be a problem, but has been a problem forever. Iess can
spread faster, truth can move at the same speed. Which is the best you can
hope for.

What happens to traditions and Reputations and (how) are they replaced?
Answer: who is na journalist and hopw did they get the rights and
priviledges to do that? (Griffin). It’s no different from now (trippi)It’s
not an eitgher or situation.

Creativity is happening at the edge of the network – group think can lead
to mediocrity.
What happens when your friends all discuss a restaurant to go to and
everyone gts an opinion? The food isn’t that great, no one hates it, but no
one loves it either.

My Comments

Mociology, which was part of the event, is a made up word with seemingly no
use outside the event. It is simply a snazzy buzzword for the “stuff
happening with mobile communications”. It’s shorter at least. While the
three speakers agreed that things were changing, there was disagreement
over how much.

While Jim and Joe spoke well on their areas, much of Martin’s time
was voxpops from people on the street in Manchester answering “what would
you like to do with your mobile phone besides talk and text”. While the
revelation that people wouldn’t pay much for music on their phones was
surpring from someone involved in a company selling ringtones, it seemed
out of place against the substantive comments of the other two.

While the chair was insightful in some comments relating to
his area of expertise, there was an intake of breath from
some of the techies in the room when he asked Joe “how did
you set up a blog?”.

Update: having heard from the chair by email, it appears
that the question was intended to ask how the blog was set up in a social
sense – how people got started talking about it and using it
- rather than the mechanics of the technical operations.

Overall, the event was worth the 6+ hours on a train that it
took to get there and back. While it was patchy in places,
it was worth attending.

Welcome to DisruptiveProactivity.com. A new site to bring together the e-democracy stuff I do, and to blather, sometimes coherently, about the stuff done by others.