Skip to content

GSoC 09 weekly report #3: 05/06 – 12/06

June 16, 2009

After my exams, I continue working. I’ve prepared a proposal to the driver’s implementation. Totally in Request-For-Comments state, of course

EDIT 2009-06-17:  Some errors corrected. New version of the document

GDAL WKT Raster driver specification proposal v0.2 (PDF 629KB)
GDAL WKT Raster driver specification proposal v0.1 (PDF 633KB)

This week, I’ll really start coding following the principles of the proposal.

Advertisements
7 Comments leave one →
  1. June 16, 2009 12:23 pm

    Great job!

    One note, name of overviews metadata table has been defined as RASTER_OVERVIEWS.

    p.s. it would be easier to read the report as a plain HTML blog post

  2. Even Rouault permalink
    June 16, 2009 9:35 pm

    A few points :

    1) You probably don’t need to subclass GDALDriver. 99% of the drivers just instanciate it, set a few metadata to describe it and provide a pointer for the pfnOpen, pfnIdentify methods (and pfnCreate, pfnCreateCopy if you want to deal with creating datasets)

    2) For the matching between GDAL Data types and WKT raster types, when you don’t have exact matching, generally you select the GDAL data type just wide enough to hold the WKT raster one and set appropriate metadata so the client application can interpret it correctly. Here’s my proposal :

    1BB –> GDT_Byte and set the NBITS=1 metadata item of the “IMAGE_STRUCTURE” domain on the raster band. See frmts/gtiff/geotiff.cpp for example. If you want to be consistant with the GeoTIFF driver, you would define a palette with 0 mapped to (0,0,0) and 1 to (255,255,255). But that’s a minor detail.
    2BUI –> GDT_Byte, with NBITS=2 (“IMAGE_STRUCTURE” domain)
    4BUI –> GDT_Byte, with NBITS=4 (“IMAGE_STRUCTURE” domain)
    8BSI –> GDT_Byte, with PIXELTYPE=SIGNEDBYTE (“IMAGE_STRUCTURE” domain). Done in frmts/gtiff/geotiff.cpp
    8BUI –> GDT_Byte
    16BSI –> GDT_Int16
    16BUI –> GDT_UInt16
    32BSI –> GDT_Int32
    32BUI –> GDT_UInt32
    16BF –> GDT_Float32 (probably no need to set a specific attribute)
    32BF –> GDT_Float32
    64BF –> GDT_Float64 (you’ve mapped it to CFloat64 in your PDF. probably a typo)

    3) As you noted, the PGCHIP driver is pretty experimental and it’s unmaintained. If you do well with your project, I think we’ll just get rid of it 😉 Trying to be consistant as much as possible with the OGR PG syntax for opening the DB is certainly a good idea. (Not sure if it can help you, but for the sake of completeness, I wanted to mention the doc of GeoRaster driver – http://gdal.org/frmt_georaster.html -, which has to my understanding a scope similar to yours with WKT raster)

    4) Probably because I haven’t read enough about WKT raster project itself, but you mentionned an interaction between GDAL and OGR which I don’t understand how this will work : “And these new type of GDAL data will be used to perform seamless operations with VECTOR DATA (Geometry type, in OGR-Syntax), from OGR library.” and then “This new abstract Raster objects will be used together with the OGR Geometry objects in functions like ST_Intersection or ST_Union. Then, you can perform similar operations with both”.

    5) You mention SQL expressions passed in the Open method that would follow the OGR SQL syntax. But shouldn’t those SQL queries be directly passed to the PG server instead so that the WKT raster extensions are used instead of the OGR SQL engine ?

  3. June 17, 2009 11:24 pm

    Hello,

    Mateusz: Many thanks! And about the format change: do you know a good pdf2html conversor? I’ve tried pdftohtml (http://pdftohtml.sourceforge.net), but I don’t like the final result.

    Even: Thanks for your advices. About the 4) point, It was a mistake in the document. A mix of concepts result of the lack of sleep. Don’t take it into account :-). It has been corrected in the new version (0.2). And aobut the 5) point, I need to think a bit more before my response. Thanks again.

  4. June 18, 2009 6:18 pm

    Good luck on Exams!

    Here’s an installation on OSX.

    ch1=# select * from rpt_versions;
    select * from rpt_versions;
    -[ RECORD 1 ]——-+—————————————————————————————————————————–
    current_database | ch1
    current_user | hamannj
    postgres_version | PostgreSQL 8.3.5 on i386-apple-darwin9.6.0, compiled by GCC i686-apple-darwin9-gcc-4.0.1 (GCC) 4.0.1 (Apple Inc. build 5465)
    postgis_version | POSTGIS=”1.4.0SVN” GEOS=”3.0.0-CAPI-1.4.1″ PROJ=”Rel. 4.6.1, 21 August 2008″ USE_STATS
    wkt_version | : 3949 $
    wkt_date | 2009-06-12 07:55:13
    r_version | R version 2.8.1 (2008-12-22)

    I’m now trying to figure out how to get an image from:

    http://rapidfire.sci.gsfc.nasa.gov/subsets/?subset=USA1.2009168.terra.250m

    into a database using

    rufus:scripts hamannj$ python gdal2wktraster.py
    Traceback (most recent call last):
    File “gdal2wktraster.py”, line 32, in
    from osgeo import gdal
    ImportError: No module named osgeo
    rufus:scripts hamannj$

    which led me to:

    http://trac.osgeo.org/gdal/wiki/GdalOgrInPython

    which got me to:

    from osgeo import gdal
    from osgeo import osr
    import osgeo.gdalconst as gdalc
    from optparse import OptionParser, OptionGroup
    import binascii
    import glob
    import math
    import sys
    import os

    which looks like Python code.

    Which is great, but I’m not sure what to do now, so can I ask has anyone reported this yet and where do we report builds? Can I do anything to help? I’m working on a geospatial agent for forests. I can’t contribute money, but I can contribute an hour here and there.

    Thanks for what you’ve done so far. Its Great!
    Jeff

  5. Tamas Szekeres permalink
    June 19, 2009 1:47 am

    Jorge,

    Just a few comments,

    The document looks like a good starting point, many of the things we came up with have been considered/mentioned.
    I’d raise the following things which should also be considered:

    1, You should describe your final connection string syntax. This will be added to the GDAL driver description when this work will be incorporated in gdal.

    2, SQL in the connection string would be dispatched directly to the driver (as the sql will result rasters), we won’t use OGR SQL here by any means. The driver will combine the returned rasters in RasterIO into the user supplied data buffer.

    3, We should also consider how the tiles will finally be handled when fetching the raster data (either when regular_blocking is set or not set in the RASTER_COLUMNS metadata table) How the tiles (table rows) should be indexed?

    4, I think we can assume homogeneous rasters in the same table, the driver should report an error if the returned image doesn’t match with the band/pixel types enumerated in the metadata table

    5, I’m still unsure about the desired raster format between the driver and the data source, I guess this should be configurable by the user somehow. (Not sure about the postgis native format either) Encoding to jpeg/png etc seems to bring in some additional overhead when fetching the data

    6, I couldn’t follow exactly what you mean by the factory class what was the objective? Isn’t just we should keep one to one relation between the dataset and the corresponding data table?

    In addition to think about these issues I think you should start coding the skeleton of the driver and many things will be more obvious just by creating the corresponding implementation.

    Do you already have some kind of SVN access or should I provide a repository for you in my development server. It looks like having a sandbox access in osgeo will not be a quick realization target 😉

    Tamas

  6. June 19, 2009 5:32 pm

    Hello,

    Jeff: Thanks for your words! And thanks for offering your help. The WKT Raster type in PostGIS is, from now, a beta project. The script gdal2wktraster, from Mateusz Loskot, is the first tool for loading images as raster in PostGIS, but it’s under development. My work is most focused in developing a read-only driver (written in C/C++) that allows reading this new PostGIS type and transforming it into a well known raster format (maybe PNG, maybe JPEG… look at Tamas’ comment under yours).

    Anyway, if you want to help, I encourage you to write to gdal-dev list(gdal-dev@lists.osgeo.org) and/or postgis-devel list (postgis-devel@postgis.refractions.net). And maybe you should look for “WKT Raster” in Postgis lists. For example: http://www.nabble.com/forum/Search.jtp?forum=1221&local=y&query=“wkt+raster”

    Any comment or idea about this new PostGIS type and about the GDAL driver will be appreciated 🙂

    Tamas: Thanks for your comments. I answer them:

    1: Ok.
    2: Understood. Is the driver, not the dataset, the one that will deal with the SQL sentences.
    3: Something to think on 🙂
    4: Ok. This will be one of the driver’s integrity checkings
    5: Maybe the user can provide a flag to select the desired format (limited to JPEG and/or PNG, at first), and choose WKB as default format (well documented, open gis standard…), if none is selected. But I’ll follow your indications on this, of course.
    6: My point is: the new WKT Raster type will can be a single image, a tiled image, a regulary/irregulary tiled raster coverage, etc. These are different “shapes” for the same GDAL concept: a new type of raster. Using a Factory pattern, we can delegate the “low-level format issues” to the subclasses of a generic “raster” class, and simply create a new instance of this generic class. Am I wrong?, too much complicated, maybe?

    About the idea of keeping one to one relation between the dataset and the corresponding data table, maybe the driver could manage this with a Singleton pattern…

    And thanks for your offer about SVN access. I’ve configured one in this domain (http://gis4free.org/gdal_wktraster/). I’m going to use it, but I’ll write you if I have problems. I’ve created a user for you, and for Frank. I’ll provide you the passwords as soon as I import the code.

    Best regards,
    Jorge

  7. June 20, 2009 11:28 am

    OK. The storage format for wkt raster is always the same (https://svn.osgeo.org/postgis/spike/wktraster/doc/RFC1-SerializedFormat), and functions of rt_core, like raster_from_* or raster_to_* deal with internal data complexity, so is not necessary to complicate the code as I said in point 6. And the Singleton pattern is not necessary. Keep it simple 🙂

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: