GeoBase

     

    Hip Hip Hooray! http://geobase.ca/ is live and online, ready and waiting to serve up delicious free canadian geospatial data. Now for our regularily scheduled "Matt criticizes everything" broadcast

    They've cleaned up a lot of things in the last six months. Performance still sucks, but maybe that's because this is their big launch day. Everything (that I've looked at) is tiled. That means to download the geographical names for 105D you need to ask for 105D01, 105D02 etc. (and you must use the mouse for each tile. no cutting and pasting or typing allowed).

    Upon submitting you are asked for an email address to which, after a time, the ftp instructions are sent. The length of time varies according to the number of tiles requested.

    Looking at the delivery instructions, it is clear that they are using a secure transaction process and now we understand why the server is easily overloaded. You are not directed to a public ftp directory but to unique numbered location like this ftp://ftp.cits.rncan.gc.ca/pub/geobase/024343_1069262100484/ where each file also has a unique session id like 105A01_1069262100484.zip. Why a computationally expensive procedure like this is being used for free unrestricted-use data is a good question.

    If you hack the form and use wildcards, delivery instructions are sent but there is nothing in the identified folder to pickup. Looking at the URLs I'm pretty sure a script could be thrown together with relatively little effort to bypass all the grunt work and send a "deliver me everything" request.

    One wonders whether anybody will do this before the end of their big launch day. And whether the server will come down in flames or just slow like ever thickening molasses as the temperature drops (brrr! it's cold outside today!). Please people, be kind. Just because something can easily be done doesn't mean it should.

    All in all it's a very beautiful looking site but to actually use it for more than a few bite size pieces of data at a time is painful. The good news here is that the data is available and free. There is nothing in the license agreement which prevents another agency from grabbing the data and hosting it in a more usable fashion. NRCAN and CCOG should be congratulated for their efforts and thanked effusively for liberated this sizable chunk of canadian geospatial data.

    (P.S. if you are using a popup blocker you won't get treated to the interactive web-mapping interface. To launch the web map interface use http://www.geobase.ca/cwc2/cwc2.php for english and http://www.geobase.ca/cwc2/cwc2.php?TEMPLATE=geobase_fr.html for french.)

    -- MattWilkie, sometime in November 2003


     

    Hacking geobase.ca

    or: how to make geobase.ca more useful

     

    Easy: Ask someone else who has already gone through all the garbage to give, or sell, the data to you! (see end of page)

    Medium: see below

    Hard: use http://geobase.ca/ as designed

     

    Ordering Data

    Go to Data Extraction, select the data layer you are interested in, zoom in until you get to the form which has a text area box for selecting individual tiles, then:

     

  • bookmark or drag the URL to a folder (the title will be like "Geobase / Select")
  • use the undisable bookmarklet to activate the form so you can type in it
  • type or paste a comma separated list of the tiles you want (the total number of characters has to be less than 1850) and press the Submit button
  • Choose the format and language options and Save Options. Bookmark the resultant URL ("Geobase / Confirmation"). Double check the list of selected tiles.
  • Slap Delivery, you will be presented with license terms, presuming you accept them punch I Agree
  • Enter an email address. Don't use a fake one as delivery instructions are sent to this address. Bookmark this URL before tapping Delivery
  • After a lengthy wait, perhaps 5 minutes, you should get a confirmation page with an order number and the list of tiles you selected. The title is "Geobase / Order Completed". Double check the list of selected tiles.
  • Wait for up to several hours to recieve an email with ftp instructions for collecting your data.
  • If, as is likely, you want more data than can scrunch into the 1850 character limit, use the bookmarks to shorten the process:

    1. Bookmark 1: Undisable > Paste > Submit
    2. After page loads use Bookmark 2 to save your format and language options
    3. After page loads use Bookmark 3 to skip license page; slap delivery
    4. wait for confirmation number

    We need to use bookmarks because 'sessionid' (and a temporary cookie I think) is used to remember your selections. However, thankfully, it appears that once an order is completed the same sessionid can be used over and over again.

    URLS

     

  • http://geobase.ca/
  • Geobase Data Extraction
  • 'undisable' read only forms bookmarklet
  •  

    Downloading

    You will want an ftp client capable of accepting a long list of URLs to download. If you don't have a favourite try wget or curl (part of Cygwin) for the command line and File Zilla for gui-ness.

     

    Post Download

    Okay, so now you have a thousand or so files with a bunch of extra garbage in their names to work with. Windows users can grab a utility like wildcard renamer and shorten 107N03_1069375429328.zip to 107N03_dem.zip in a few seconds.

    I leave it up to you to append or mosaic all those tiles into larger mealsized chunks, but here a few snippets to get you started (assumes a windows machine with cygwin installed):

    Command to unpack distribution zips all at once:

    D:\geobase\data\>for %%a in (..\distrib\cded\*.zip) do unzip %%a
    
    Now use wildcard renamer to strip all the garbage from the names. If you are like me and now have almost no disk space left, gzip everything with:
    D:\geobase\data\> gzip *.dem
    

    Now we are ready to bring into ArcInfo (assumes TIN license; if you don't have Arc or TIN see below for other options):

    D:\geobase\data\> for %a in (*.gz) do (
       gzip --decompress %a
       rename %~na %~na.dem
       arc demlattice %~na.dem %~na usgs # int
       gzip %~na.dem
       )
    

    And now we mosaic all the teeny morsels into real meals:

    /* short ArcInfo AML to create 1:250k quads out of 32 50k half pieces
    /* e.g. 094a01_e, 094a01_w, 094a02_e ...
    GRID
    &do quad &list a b c d e f g h i j k l m n o p
       &do tile &list 094 104 114    /* <-- change to suit
          &if [exist %tile%%quad%* -grid] &then &do
             &ty ...%tile%%quad%
             dem_%tile%%quad% = mosaic([listfile %tile%%quad%* -grid])
             /* if you are really brave, uncomment the following
             /* line to delete the source tiles:
             /* kill (![listfile %tile%%quad%* - grid]!) all
             &end
       &end
    &end
    QUIT
    

     

    without ArcInfo or TIN

    VirtualTerrainProject VTBuilder supports drag'n'drop. Just grab a bunch of .zip files (no need to unzip first!) and drop them onto the main window. (Also works for Arc grids by the way: recursive search for hdr.adf and drop those instead). Set your Extract Area to Extents, or draw a box, and then export as .bt file (no other export options available. yet).

    If you have OpenEV installed open a bunch of the .dem files (no drag'n'drop yet I'm afraid) and File > Export to the format of your choice. For a more automated route, use gdal_translate from the OpenEV/bin directory and substite gdal_translate for the arc demlattice line in the example above.

    If you have the complete gdal suite installed with python enabled you can use the gdal_merge.py script to duplicate the mosaicking step outlined above.

     

    The End Results

    As some of you know MattWilkie downloaded, projected and mosaicked the 1:50,000 digital elevation models for Northern BC (NTS 094*, 104*, 114*) available on http://geobase.ca/ .These are now online for whoever wants 'em at ftp://ftp.geomaticsyukon.ca/DEMs/NorthBC/ -- thank you Geomatics Yukon for setting aside ~2gb!

    Update, Sept 2007: The rest of Canada available now too, see http://sydney.freeearthfoundation.com/mattwilkie/draft/canada_50k_dem. Thank you Adam Nowaki and the Free Earth Foundation for hosting it.

    There is no support for this data, it's up to you to figure out how to load and use it. If you're stuck you can still email Matt but it may take some time before any response is forthcoming.

    The quality of the 50k DEMs is widely variable. Some are just awesome, clearly having much more detail than is possible from 50k data alone -- I'm pretty sure they are derived from the 20k TRIM. Others are, well, still clearly derived from TRIM but also clearly wrong: significant north-south smearing patterns and occasional sunken countours. Use the shaded relief images to get a general indication of which areas are problematic.

    Also downloaded are the 'geographic placenames' and 'primary vertical benchmarks' for YT but nothing been done with them yet. Maybe in January but Matthew a big project starting then so it could take awhile to process. You don't have to wait for him though, you have just read the outline of a process to make Geobase more useful, so get going!

     

     

    I Attachment Action Size Date Who Comment
    batcmd gen50k-list.cmd manage 0.4 K 2003-11-26 - 16:07 TWikiGuest generate list of 50k tiles
    txttxt list-all-50k.txt manage 21.0 K 2003-11-26 - 16:06 TWikiGuest all YK 50k NTS tiles, in lines of less 1800 char

     

        Send feedback