geobloggers

In a continuation of my post the other day and because it’s Friday night, it’s time for a stupid, overly complex but satisfyingly fun hack.

Why is it stupid? Because I’m going to be using JavaScript to load data from Yahoo! Pipes which is loading data from triplr.org which is in turn going to load data from geonames.org. Giving us three points of failure as data flies around the web from service to service.

Why is it good? Because we’ll learn a few tricks about how to do stuff with just JavaScript and no yucky backend stuff like PHP. And these types of experimental services (or similar services) will get more stable in the future so the points of failure become less risky.

Ok, so here’s the recap;

GeoNames.org gives us information about a gazillion places around the world, we’re going to use the Golden Gate Bridge as an example. Here’s the GeoNames page for Golden Gate Bridge…

http://www.geonames.org/5352844/golden-gate-bridge.html

You can get away with knocking the last path element from the URL to get just this …

http://www.geonames.org/5352844/

Great! But I can’t use that with JavaScript yet. Let’s dig a little deeper. GeoNames is kind enough to also spit out information about Golden Gate Bridge in lovely/horrid Semantic Web RDF format, as a bonus you can get nearby stuff too. You may need to do a view source on the following two pages to see what’s going on…

http://www.geonames.org/5352844/about.rdf
http://www.geonames.org/5352844/nearby.rdf

Good, RDF. However I prefer messing around in triple space. Lets use triplr.org, which takes data in, in all sorts of formats, and spits it back out again, as triples, in all sorts of formats. An easy way to see what’s going on is to tell triplr to output the RDF above in HTML. We do this by using the right format URL thing on triplr.org to load in the target URL we want to parse, here we go…

http://triplr.org/html/http://www.geonames.org/5352844/about.rdf

Well, that’s handy for us humans to see what’s going on. But I want to use it with JavaScript damnit! Bonus, triplr also gives us the results in JSON. If you ask it nicely it’ll even make it pretty for you. Instead of using http://triplr.org/html/my_url you can use http://triplr.org/json;pretty/my_url. Triplr will give us back a JSON “triples” object containing an array of other objects, which sounds nice so lets do that …

http://triplr.org/json;pretty/http://www.geonames.org/5352844/about.rdf

Lovely. Unfortunately we can’t use that JSON object as is, within our JavaScript. Here’s the code we’d normally use in JavaScript to grab some external JSON…

var s = document.createElement(’script’);
s.src = ‘http://triplr.org/json;pretty/http://www.geonames.org/5352844/about.rdf‘;
document.getElementsByTagName(’head’)[0].appendChild(s);

But if we did that we’d get an error, what we need is the JSON object returned as a parameter in a function call. This page on flickr, is as good as any page, at explaining what I mean …

http://www.flickr.com/services/api/response.json.html

What we need is one last step to wrap a function call around the JSON, and this is where we pull in Yahoo! Pipes. I build a Pipe to take the JSON output from triplr.org and convert it to, errr, well JSON. But! Pipes allows you to add a callback function to the JSON and that’s what I’m going to do. Here’s my lovely Pipe over at Yahoo!

http://pipes.yahoo.com/pipes/pipe.edit?_id=EsCFhUff2xGWtBTQ1fC6Jw (view the pipes page)

Here’s what I’m asking it to do; ask for the user to enter a feature number, with a default value. Build a URL using that feature number, fetch the URL and output the result. From top to bottom…

1. When running the Pipe, prompt the user for a Feature Number. We actually won’t do this, but it means the feature_number will be a parameter that we can change on the URL that Yahoo! Pipes gives us at the end. We feed this number into the URL builder.
2. The feature number is the first element of the geonames URL. So our base element is “http://triplr.org/json;pretty/http://sws.geonames.org”, then our feature number is injected and we finish off the whole thing with the “about.rdf”, giving us the full URL of “http://triplr.org/json;pretty/http://sws.geonames.org/5352844/about.rdf“.
3. Pipes fetches the data, but it always wants to chuck out a list of some kind at the end of the day, so it’s asking us the path to the list. Looking at the pretty JSON object given back to us by triplr we can see that the root path is “triples”.
4. Pipes output, this is the list of objects it found. You can see the start of the list in the Debugger Panel at the bottom of the image above.

Now the next step is to run the Pipe, this is what the screen looks like …

http://pipes.yahoo.com/pipes/pipe.info?feature_number=5352844&_id=kp5v7Ubf2xGACX7J1fC6Jw&_run=1&=Run+Pipe

You can see where you can enter the geonames.org feature number at the top, and the link to “Get as JSON” down at the bottom. You’ll need to scroll down past a list of “Untitled” to get to it. Clicking the JSON link gives us two things, a result like this …

Which is still just JSON that we can’t use. However more importantly we also get this URL, broken up to make it more readable…

http://pipes.yahoo.com/pipes/pipe.run
?feature_number=5352844
&_id=kp5v7Ubf2xGACX7J1fC6Jw
&_run=1
&_render=json

Now, finally we can do the magic last step, we add the parameter “_callback=parse_json” to the URL. If we use the following …

http://pipes.yahoo.com/pipes/pipe.run
?feature_number=5352844
&_id=kp5v7Ubf2xGACX7J1fC6Jw
&_run=1
&_render=json
&_callback=parse_json

We get our JSON back, but wrapped in a function call. You can see it right at the start of the following screenshot parse_json( … json object … ) …

Phew, there we go. From the original RDF from geonames.org we end up with a JSON fragment that can finally be loaded into JavaScript. The following code is all we need.

<script type=”text/javascript”>

function parse_json(json) {
alert(’Woo, I have JSON: ‘ + json);
}

var s = document.createElement(’script’);
s.src = ‘http://pipes.yahoo.com/pipes/pipe.run?’+
‘&feature_number=5352844′+
‘&_id=kp5v7Ubf2xGACX7J1fC6Jw’+
‘&_run=1′+
‘&_render=json’+
‘&_callback=parse_json’;
document.getElementsByTagName(’head’)[0].appendChild(s);

</script>

However, caveat, getting back to why this is a slightly stupid hack; what we’re asking JavaScript to do here is load in the script provided by Yahoo! Pipes. Which is loading data provided by triplr.org, which itself is being asked to get the data from geonames.org. We go to Pipes, which goes to triplr, which goes to geonames.org, which sends data back to triplr which sends data back to Pipes that then sends it back to us.

Pipes itself gives up if it takes longer than 30 seconds to get anything back, triplr.org is new and goodness knows how GeoNames is doing at any moment in time.

On the plus side, the above demonstrates how to use Pipes to turn stuff into JavaScript readable JSON.

Let’s play with some JavaScript then, just to prove the point. I’ve also set up another Pipe to grab “nearby” stuff from geonames. You can find it here …

http://pipes.yahoo.com/pipes/pipe.info?_id=EsCFhUff2xGWtBTQ1fC6Jw

Here’s a screen shot of the javascript in action (I’m using FireBug to help) …

Over in the bottom right panel, you can see the json object as returned from Pipes. To get to the triples I can use the dot notation json.value.items and then loop though the array. Because everything is in semantic web goodness I can start to fish out the details that I want.

For example if I want the longitude I need to look for the item object who’s predicate value is “http://www.w3.org/2003/01/geo/wgs84_pos#long”. Once I’ve found that item I look at the object value.

This is the great advantage of using triplr, any RDF it parses that has geo information in (or any correctly marked up information for that matter), in the correct format will get turned into something that we know we can find.

In the original RDF file geonames.org has this defined at the top ‘xmlns:wgs84_pos=”http://www.w3.org/2003/01/geo/wgs84_pos#”‘ and the location nodes further down is like this ‘<wgs84_pos:long>-122.4788612</wgs84_pos:long>’. When correctly marked up it means we can loop through any collections of “items” we get back from Pipes hunting for the things we hope are there.

Anyway, it’s late, so here’s the page where it’s all working*, in a very messy, hacky type of way. But a handy rapid launch point for figuring out what’s going on, view source is your friend …

http://www.geobloggers.com/json_demo.html

*Well when I say working, I mean may be working as longs as geonames.org, triplr.org and Yahoo! Pipes are all working :)

And that’s how we can load (in theory) and parse a RDF document in JavaScript from a remote server.