Monthly Archives: September 2014

Foss4g 2014: The good, the bad and the beers

After the closing-party last night I am in a post-foss4g-mood. That is, I’m hung over and over-stimulated at the same time. So this should be a perfect time to reflect on the conference. TLDR: It was great, going to foss4g feels like coming home!

So, we (that is, Alexander and me) arrived in Portland by plane from Norway last saturday, and found our “condo” rented through AirBnB. Note to self: rent a place, skip the hotels, much better! The first days we spent drinking beers, shopping and enjoying the city. Tuesday was JsGeo-day: that is a whole day filled with two things I really enjoy: Geographic data and Javascript. Several good talks, interesting remarks and a real sense of belonging.

What is actually quite cool about an event such as JsGeo is that in my day-to-day life I share the interest with about 5-6 people, but all of a sudden I am in a room with about 100 people that are really into the same stuff. Also, seeing and hearing authors of frameworks I use daily is quite something. So: JsGeo was absolutely worth it, including the after-conference JsGeoBeers at Rogue Halls. Another interesting observation is that projections are regarded as so hard, guess that shows that more and more of the developers working with geo has no formal background in the field, and that the widespread use of web mercator have made people “lazy”?

Wednesday was the start of the actual Foss4g-conference, and a bit tired we found our way to the gigantic Oregon Convention Centre, and missed the introductory talks. What we did not miss was the opening keynote by Mike Bostock, known for the d3-libary. While I personally never have managed to wrap my head around d3 I really think Bostock hit a note in the audience. I’m not sure if it was the really great projections-work or his thoughts on tool-making or the combination that caught on, but his keynote was referenced frequently by other speakers throughout the conference.

One of my main takeaways from the overall conference vibe is that foss4g is turning more into a geo-tech conference than a strictly open source geo conference, but that might be because all the interesting stuff happens in opensource? Another observation is that while we still have the stable, cross-company developed products such as PostGIS, OpenLayers, Mapserver and GeoServer there are some new players on the field that are more in the services for pay, code for free-area. Wether this is a good thing or not can of course be debated, but as long as this results in great, open source code for all to use one can’t really complain.

For technology I really think JavaScript got lots of attention this year, as well as vector tiles and web stuff in general. Also nice to see so many talks on cartography and the related topic of design. These observations may also be explained by the fact that they are things that interest me, so I naturally gravitated towards these talks. While on the topic of talks, I’ve given up counting how many times I had to think long and hard about what track to choose, with 8 tracks in paralell and so many promising talks things do get hard. I really look forward to the videos beeing uploaded, there are lots of things to catch up on.

I’m not really going to go through all the talks I saw, but I have to mention some that really impressed me. I’ve started to realize that delivering a great talk is really hard, 25 minutes is not a lot of time, so limiting scope is important. Knowing your audience is also importand, although this is really hard. Nevertheless, the majority of the talks I attended where good, focused and thought me something. While it may seem unfair to mention only some of the talks I really think some deserves a special mention:

Fiona and Rasterio: Data Access for Python Programmers and Future Python Programmers by Sean Gillies was the first talk I saw after the opening keynote and this really struck a chord with me. Seans intent of doing GIS the pythonic way really reflects how I think code should be written.

TileMill and the Tower of Prince Henry, Reversed by Eric Theise was a talk I chose because it sounded a bit strange, and by God it was. The strangest talk I’ve seen, but in a good way. I can’t really describe this talk, neither the concept, but suffice to say he started out referencingart films from the seventies and ended up proposing a rather cool idea of “hearding” the user through a web map, limiting choice and the need for large tile sets. Really hope that this idea gets somewhere, I’m tempted to try to implement some of his ideas in OpenLayers 3!

“Sliding” datasets together for more automated map tracing by Paul Mach from Strava was excellent in that it had a limited focus on a really great idea implemented in a really impressive way. OSMers should really look at this guys work, there is seroius potential for saving time here. The main idea is fitting (or sliding) existing geometries in order to “merge” or correct them.

Cartography from code…? by Barend Köbben started off the last day of the conference for me, and what impressed me was the “code as a tool”-mindset of a die-hard cartographer. I really think this is needed, cartography hasn’t really taken the front-seat in web mapping, but I think cartography will be even more important in the future.

projections in web browsers are terrible and you should be ashamed of yourself Calvin Metcalf was by far the funniest talk of the conference, continuing the underlying theme of “projections are hard” that started at JsGeo, Calvins bold, fun and “no-filter” way of presenting made me laugh hard several times, while also conveying an important message that seems to me to be even more important in the states, where every state has it’s own “projection”. Use WGS84 geographic coordinates for data exchange and web mercator unless there are valid reasons not to was my main takeaway.

While these talks are the ones that really impressed me there was, as I said, really an abundance of great talks. One thing I did notice was that several talks was a bit america-centered, assuming everyone has the same background as americans when it comes to data formats, ways of doing things and the like, but since we are in USA this is maybe rather natural?

So, the “best” have been covered in great detail, what about the bad? Well, I do not really have that much to say when it comes to “bad”, I feel that last year maybe was better at getting a social context, with a smaller venue, more joint sessions and people mostly living on campus. Also, the amount of entertainment was considerably less this year, I really missed something like the festival of the spoken nerd from last year. Apart from that I don’t think there was anything bad to write home about.

And the beer: 10 excellent breweries and bars visited, 50 different beers tasted (in smaller or larger quantities): Portland really is beer geek heaven! With Foss4g in town is really was like beeing in paradise for me for a week!

Debunking map myth #8: Scale numbers works on a screen

So, I’m obviously not going to debunk my map myths in the order they are stated, but in the order I’ve defined as “what Atle feels like writing about at a given moment”.

That brings us to myth 8, which I guess is more a myth among people buying web map solutions (and usually with a GIS-background). But then again, there are apparent solution to this problem, and there are web maps that seems to have solved it, so I guess there are programmers that believe that this is possible as well. Hence: a myth about maps (and web maps to be specific) worth writing a couple of words about.

But first, let us properly define what this myth is about: Scale (or scale numbers). To quote Wikipedia on “Scale”:

The scale of a map is the ratio of a distance on the map to the corresponding distance on the ground.

Or, as I first heard the concept of scale described when I got into the (weird) sport of Orienteering. A map scale of 1: 50.000 means that 1 cm on your printed map equals 50.000 cm (or 500 meters) in the “real world”. What implications does this have? The first, and most important one is that there is a defined relationship between the real world and the map. In other words: you can bring out your ruler, measure a distance on the map (say 10 cm), check the map scale (say 1:50.000) and then figure out that you still have 5000 meters (or 5 km) left to walk before you can rest for the day.

This relationship between the map and the real world is ingrained with people that work with maps, and naturally they expect that this relationship holds for maps on a computer screen as well. My statement is that this is in fact a myth and a falsehood. And, it seems like the rest of the world agrees, when did you last see a scale printed on a web map? I had to dig a bit and found the previous version of the map client from the Norwegian Mapping Authority as an example.

scale_line

Look at the screenshot above. (“Målestokk” is scale in Norwegian). If we apply the concept of scale we know from printed maps the following principle could be used:

  • Take out your ruler
  • Measure a distance on the screen (say the distance Oslo-Trondheim, on my screen this is 5cm)
  • Note the scale (1:3.838.112)
  • Compute the distance using the method above
  • This gives a distance Trondheim – Oslo of approx 192 km
  • The measure-tool on the pages says the distance is approx 380 km

So, if we trust the measure-tool there is clearly something wrong with the scale on my screen. Go ahead and try the same on your screen, and I will almost guarantee that you won’t get the same result as me. Why? Because pixels does not map to actual, physical, length. Several factors come into play, most significant is screen resolution. Mapping libraries such as OpenLayers assumes a DPI of 72, because this was determined to be a standard in some day. Later recommendations is to use 96, but you still won’t match every users screen resolution. Think of mobile screens and retina screens for example.

Thus: the notion that map scale works on a screen in the same way as on a printed map is flawed. So, what to do? One thing that will work is to display a scale-bar (which you can see on the screenshot above). This works, because the relationship is the same, so if I measure how may centimeters the 100 km bar on the screenshot is on my screen (about 1.3 cm) I can get the scale: 1: 7692307.7. This translates to a distance Oslo – Trondheim of about 384.615 km, which is (given all the error sources) in the ballpark of a correct answer.

So, skip fixed scale numbers, and use a scale bar. Even simpler, do not talk about scale at all, but operate with zoom levels such as “country”, “city”, and “street”. In this way you give no impression that your web map has a scale in the same way as a paper map.

So, with these calculations I think I’ve debunked this myth. I’m aware that I’m a bit hazy on the cm – pixel – dpi – relationship, but that’s intended. I’m no expert on these things, but I do hope that some knowledgeable reader can fill in some gaps for me?

Debunking map myth #1: All coordinates are in “Latitude/Longitude”

Yesterday I published a post called Falsehoods programmers believe about maps, where I listed 18 falsehoods I believe or have seen programmers believe about maps and spatial data.

What I did not do was provide any backing for my claims that these are in fact falsehoods, and after this was pointed out in the comments I figured “here’s material for some follow-up posts”. I’m not sure I’ll go through all the falsehoods, and it might be that I have to revise some of my claims, but I’ll start with the first falsehood in this post.

So: the first falsehood is: “All coordinates are in “Latitude/Longitude””. Why isn’t it?

TLDR: look at a search for coordinate systems for Norway: “Found 247 valid records”. This is far more than 1, so case closed.

In depth: We separate between geographic coordinate systems, which is (to simplify a bit) angles of latitude and longitude on an ellipsoid that represents the earth. These coordinate systems are also known as “unprojected coordinate systems”. What makes things even more difficult is that there is several reference ellipsoid models used (specified by ellipsoid parameters a and f) and also different datums (specifies how the actual earth is aligned with the ellipsoid).

Thus: given pair of longitude/latitude coordinates, there are several options for ellipsoid and datums. True enough: when people refer to lat/lon they usually refer to the ellipsoid/datum-combination most widely used: WGS84. This is the system used by GPS, by Google Maps and in most cases where “laypeople” refer to “latlon”. Still, there are a lot of other datums (such as ED50, still used by the Norwegian petroleum industry).

In addition to the unprojected geographic coordinate systems we also have a bunch of projected coordinate systems. These usually have meters (or feet for the strange americans) as units, and one can actually use (with some minor exceptions) the Pythagorean theorem to calculate distances between coordinates. The downside with projected coordinate systems is that going from a 3- to 2D-representation _will_ introduce some errors (see Mercator). One common solution is to make a projected coordinate system valid for only “small” areas of the earth (such as “norway”). There are a lot of techniques for projecting a map, but the most commonly used is the Universal Transverse Mercator (UTM). This is not a single projection, but a series of zones.

Zones 32N, 33N and 35N covers Norway, and for web mapping purposes it’s ok to use zone 33N to cover the whole of Norway (there will be errors in the edges, but these are negligible for purposes of screen display). The UTM projection is usually used in combination with the WGS84 datum, but can also be used with other datums.

So: in conclusion: there are a lot of ways to express where on the earth you are, and when given a “latlon”-pair you cannot be sure what coordinate system it’s in (although a fair guess is WGS84)

Falsehoods programmers believe about maps

Ok, I’m probably not the first one to write a post about falsehoods programmers believe about maps, but I still remember reading “Falsehoods programmers believe about names” and “Falsehoods programmers believe about time“, and thinking “there should be a post like that about maps”. So, go ahead, read those two posts to get to know the style of a post like this and then read on here.

  1. All coordinates are in “Latitude/Longitude” (Why is this false?)
  2. If you know the “Latitude/Longitude” you can be certain of exactly where you are
  3. “Spatial is special”
  4. The earth is round (Why is this false?)
  5. The earth is ellipsoidal (Why is this false?)
  6. The GPS-satellites know where I am
  7. There is a single, right, map projection
  8. Scale numbers works on a screen (Why is this false?)
  9. There are no good alternatives to Google Maps
  10. Web Mercator works for all purposes
  11. The shortest path between two points is a straight line
  12. All programmers agree on the ordering of latitude and longitude pairs
  13. Given a paper map I can always digitize and georeference it
  14. My background map will be better if I cache it, always!
  15. GIS software is always expensive
  16. The whole world is mapped, thouroughly
  17. Looking up a street address and get it’s position is easy
  18. Offline maps is as easy as Goole Maps

That’s what I’ve come up with this far, I might fill in something here when I think about it. Comments? Additions? Any errors? Want citations? Well, the comment section is open, so go ahead!

Lightweight coordinate transformations in JavaScript

We (as in: people that work with geospatial JavaScript) have all been there: We have coordinates as “latlons” (i.e. Geographic coordinates in WGS84), but we need to transform them to some other coordinate system (as a Norwegian: replace “some other” with UTM Zone 33N). Ok, that shouldn’t be that difficult, just include proj4.js. It’s a mere 73.275 kb, and handles almost all coordinate transformations (given that you know their definitions).

But, sometime you just care about one single transformation (say: EPSG:4326 to EPSG:32633), and you are really concerned about file size (maybe you are targeting a mobile site), because size does matter and you feel that proj4.js weighs in a bit too heavy. Well, that was what Bjørn Sandvik thought yesterday:

Then I got around to thinkning: how hard can this be? After a bit of thinking I realized that, yes, this is actually rather complicated in terms of maths. But, after a bit more thinking, I remembered this project from a couple of years ago (blog post in Norwegian). Basically, my old department at NTNU had some fortran code from around 1990 written by professor Jon Holsen in Fortran.

A couple of years ago I ported that code to JavaScript (on GitHub), but did not think much more about it (although I heard reports that the department has used it some).

After looking at old notes and books on coordinate transformations it dawned on me that Holsen.js has a “bl_to_xy”-function, which transforms Geographic to planar coordinates. I then ripped out the relevant code, removed things that wasn’t needed for my simple WGS84 to UTM33 conversion, looked up the right params for UTM33N and made a gist in the middle of the night.

I did some testing against proj4js, and Bjørn did some more (reporting: “similar down to the centimetre”), and after figuring out that the original code used X for North and Y for East it seems to do the work. Result: a js file of about 100 lines, and that is rather readable lines as well. Running this code through a minifier (uglify.js) results in a 8 kb file (that is, about 10% of proj4js).

The code is tailored to transforming to one specific UTM Zone, but it should be easy to modify to work for other zones as well. Another idea is to make a similar script for transforming the other way, from UTM33 to Geographic coordinates. Holsen.js has support for this, so it shouldn’t be that difficult.

In closing: I think that small, simple scripts like this (which is small in size and does _one_ thing) is much more needed in JavaScript than in other languages. Download size matters for web apps, and the idea of one, monolithic library to solve all conceivable tasks does not work for JavaScript the way it does for Java and similar languages. Also: beeing able to provide a fellow geo-geek with working code in the course of a couple of hours is always cool! In addition: nice to see that work I did mostly for fun two years ago can be of use to someone.