Skip to content

GDAL PostGIS Raster Driver

December 9, 2010

Some people are asking for improvements and bug fixes on GDAL PostGIS Raster driver. Since r20912, there is an improved version of the driver in GDAL trunk. I’m working on it. And I know there are people making efforts to bring PostGIS Raster to some Desktop viewers by using GDAL. Many thanks!

Apart from that, I’m working on PostGIS Raster MapAlgebra. First version before January (I hope…)

WKT Raster is now PostGIS Raster

October 1, 2010

The WKT PostGIS raster code is now part of PostGIS trunk. Quick install instructions (from PostGIS sources):

  1. ./configure –with-raster
  2. make
  3. make rt-all
  4. make install
  5. make rt-install
  6. createdb <your_database>
  7. createlang plpgsql <your_database>
  8. psql -f postgis/postgis.sql -d <your_database>
  9. psql -f spatial_ref_sys.sql -d <your_database>
  10. psql -f raster/rt_pg/rtpostgis.sql -d <your_database>

UPDATE 1 (2010-10-22): With the last release (r6095) it’s an easier process:

  1. ./configure –with-raster
  2. make
  3. make install
  4. createdb <your_database>
  5. createlang plpgsql <your_database>
  6. psql -U <your_user> -f postgis/postgis.sql -d <your_database>
  7. psql -U <your user> -f spatial_ref_sys.sql -d <your_database>
  8. psql -U <your user> -f raster/rt_pg/rtpostgis.sql -d <your_database>

In case your user requires a password, add -W as psql flag, and you’ll be prompted for the password. Further information in README.postgis and README.raster files.

And of course, I’m working on GDAL PostGIS Raster driver. I’ll add the new specifications ASAP.

UPDATE 2 (2010-10-21): New version of the driver committed to repository.  Now its name is postgisraster, instead the old one: wktraster. Further information in the link above.

Comparing Oracle GeoRaster with PostGIS WKT Raster (II)

September 1, 2010

In the previous post, I installed and configured Oracle GeoRaster, and did some basic stuff: load a shapefile, load raster data, view these data and export them, using PL/SQL and Java. We compared some points I consider Oracle GeoRaster’s weak points with PostGIS WKT Raster.

Now, I’d like to reproduce the Pierre Racine’s tutorial, but using Oracle GeoRaster. Will it be possible? Let’s see it…

PREVIOUS STEPS

We’re going to use OpenJUMP, to see the results of our queries. So, we first install it. Easy task. But now, we need a plugin that allows OpenJUMP to retrieve geographic information from our Oracle Database. The plugin I used was JUMP DB Query Plugin.

The instructions are pretty clear, so, I only had to:

  1. Unzip the plugin files into the OpenJUMP lib/ext folder
  2. Download and unzip the correct Oracle JDBC driver for my Oracle Enterprise Database version (11.1.0.6.0) in the OpenJUMP lib/ext folder.
  3. Run openjump.bat

And that’s all. Ready to start.

Loading shapefiles (caribou distribution shapefiles)

I used the cariboupoint table generated in PostGIS, just as described in the tutorial. Using pgsql2shp, I dumped the table data into a shapefile.

Dumping cariboupoints table

Then, using shp2sdo again, I converted my data to SDO format:

Transforming cariboupoints SHP to SDO format

Of course, I could create the data using a PL/SQL procedure, just like Pierre did in his tutorial. But the procedure was created only for providing a shapefile to work with, in case you don’t have access to real caribou distribution data. We can use the same file.

Visualizing the caribou point in OpenJUMP

Now, from OpenJUMP, run the query select geom from cariboupoints. The result, here:

Viewing cariboupoints table from Oracle in OpenJUMP

Loading raster data

As we showed in the previous post, we need to insert 4 GeoTIFF files, covering the whole area of Spain. Specifically, 4 SRTM 90m DEM files. There are 2 more files, covering the area of Canary Islands, but they are not continuous with the rest of the data. As Oracle GeoRaster uses a one-georeference-by-layer georeference schema instead of one-georeference-by-raster one, you can’t store non-continuous raster data in the same raster data table. For the purposes of this tutorial, we only need the 4 files covering Spain area.

So, as we saw in previous post, we first reformat and reblock GeoTIFF images, to ensure they don’t have BSQ interleaving:

gdal_translate -of GTiff -a_srs epsg:4326 -anodata -32768 -co “TFW=YES” -co “INTERLEAVE=PIXEL”
-co “TILED=YES” -co "BLOCKXSIZE=50" -co "BLOCKYSIZE=50" image.tif image_new.tif

 
Options used:

  • -of Gtiff: The output format for the reformatted files. GeoTIFF again.
  • -a_srs epsg:4326: This is the srid for the data. The same than input files.
  • -anodata -32768: This is the NODATA value for bands. We can get this value using gdalinfo over the original files.
  • -co “TFW=YES”: This forces gdal_translate to create an output tfw file, with georeference data.
  • -co “INTERLEAVE=PIXEL”: Forces pixel interleaving in the output files, the only allowed interleaving schema.
  • -co “TILED=yes”: Forces the creation of stripped TIFF files.
  • -co “BLOCKXSIZE=50″, -co BLOCKYSIZE=50”: Dimensions for tiles. The same used in WKT Raster tutorial.

Further information about GeoTIFF driver for GDAL here. Now, we have to repit this with every single file we’re going to load. Once done, we can load the files in Oracle GeoRaster, using SDO_GEOR.importFrom procedure as follows:

DECLARE
geor SDO_GEORASTER;
BEGIN
-- Initialize an empty GeoRaster object into which the external image
-- is to be imported.
INSERT INTO spain_images
values( 1, 'Spain_TIFF_1', sdo_geor.init('spain_images_rdt') );
-- Import the TIFF image.
SELECT image INTO geor FROM spain_images
WHERE image_id = 1 FOR UPDATE;
sdo_geor.importFrom(geor, 'blocksize=(50,50) spatialExtent=TRUE', 'TIFF', 
'file', 'C:\orcl_tut\srtm_35_04_new.tif',
'WORLDFILE', 'FILE', 'C:\orcl_tut\srtm_35_04_new.tfw');
UPDATE spain_images SET image = geor WHERE image_id = 1;
END;

 
Three comments here:

  • The option ‘blocksize=(50,50)‘ is not really needed. As we can read in the Oracle Spatial Developer’s guide, storage parameters section, “If you specify neither blocking nor blockSize, default values are derived from the source GeoRaster object: that is, if the original data is not blocked, the data in the output GeoRaster object is by default not blocked; and if the original data is blocked, the data in the output GeoRaster object is blocked with the same blocking scheme”. As long as we specified block size when reformatting GeoTIFF files, we wouldn’t need to use that option now.
  • The option ‘spatialExtent=TRUE‘ will be necessary later. Anyway, if we omit this option, we can generate the spatial extent of the raster objects by SDO_GEOR.generateSpatialExtent procedure, once raster data are loaded. Further information on generating and setting spatial extents in the Oracle Georaster Developer’s Guide, section 3.6
  • We provide, as additional arguments, the World File, with georeference information.
Now, we can see the extent coordinates in the correct reference system:

GeoRaster spatial extent ok

As you can see, the extent of the raster is represented as a polygon in SDO_GEOMETRY format. For me, looks complicated… If you want to know more about the meaning of all fields of the SDO_GEOMETRY object, you can check it here.

What is the reason of keeping a representation format so complicated? Ok, it could be my impression. After all, I’m new in this world. Anyway, I started a thread on PostGIS devel list, asking for this. One important topic that came up in conversation is Oracle has a topological storage model. But if you want to work with this model, you must use the SDO_TOPO package. And there’s probably no one tool able to work with this model, apart from Oracle itself, of course.

Take into account, in this particular case, the SDO_GEORASTER table is composed of 4 objects, because we had 4 GeoTIFF files. So, the extent is calculated for each object, not for the whole coverage. If we want only one object, we should:

But, honestly, I didn’t try these solutions yet.

The last thing we have to do with raster data is… create and index. As we can read in Oracle GeoRaster developer’s guide, section 3.7, “the most important index you can create on a GeoRaster object is a spatial index on the spatial extent (footprint) geometry of the GeoRaster object”. And we need to take one more thing into account… According to the documentation (chapter 18), one of the prerequisites for creating an spatial index over a geometry column is “The USER_SDO_GEOM_METADATA view must contain an entry with the dimensions and coordinate boundary information for the table column to be spatially indexed”. And again, in the documentation (chapter 2, section 2.8): “Spatial users are responsible for populating these views. For each spatial column, you must insert an appropriate row into the USER_SDO_GEOM_METADATA view. Oracle Spatial ensures that the ALL_SDO_GEOM_METADATA view is also updated to reflect the rows that you insert into USER_SDO_GEOM_METADATA.” .

So, let’s do it:

DELETE FROM user_sdo_geom_metadata WHERE table_name = 'spain_images' AND 
column_name = 'IMAGE.SPATIALEXTENT';
INSERT INTO user_sdo_geom_metadata VALUES ('spain_images', 'IMAGE.SPATIALEXTENT', 
SDO_DIM_ARRAY(SDO_DIM_ELEMENT('X', -180, 180, .00000005),
SDO_DIM_ELEMENT('Y', -90, 90, .00000005)), 4326);
DROP INDEX spain_images_sidx;
CREATE INDEX spain_images_sidx ON spain_images(image.spatialExtent) INDEXTYPE IS mdsys.spatial_index;

 

Once done, our raster data is ready to work.

And WKT Raster?: All things we’ve done to load our raster data in Oracle GeoRaster and create an index, can be done in PostGIS WKT Raster by executing these two lines:

> gdal2wktraster.py" -r C:\orcl_tut\*.tif -t spain_images -s 4326 -k 50x50 -I 
-o C:\orcl_tut\srtm.sql
> psql -d tutorial01 -f C:\orcl_tut\srtm.sql

 
Of course, you need to install all the PostGIS stuff. You have an useful tutorial on how to do this here (in Portuguese, but I think it’s easy to follow. If you find it difficult, you can read WKT Raster documentation or ask in the PostGIS users list.

So far, I find too difficult the simple operation of loading GeoTIFF data in Oracle GeoRaster, comparing it with the same operation done with PostGIS WKT Raster. Of course you can always ask in Oracle Spatial forum (register required). The questions I’ve asked were quickly answered. Really great people in these forums.

Another important difference between both extensions are the indexes. Oracle GeoRaster creates spatial indexes over the footprint of raster data, and WKT Raster creates GiST indexes over the raster data itself. Just a newbie guessing: as far as I know, the operations you can do with raster data itself in Oracle GeoRaster are rather limited. Spatial operations, like intersections, can only be done with the MBR of the data, so, that’s the reason the indexes are created over the footprint. On the other hand, in PostGIS WKT Raster, you can do really heavy operations with raster data, like ST_DumpAsPolygons. Then, create an index over these raster data has more sense.

And if you want to interact with real raster data in Oracle GeoRaster? You can use the functionalities provided by SDO_GEOR package or directly attack the data blobs, using DBMS_LOB package. But this last option is a blind attack over raw binary data.

About the spatial extent of the raster data, in WKT Raster the spatial extent of a raster is calculated during the loading process, using GDAL provided raster dimensions, and calculating the georeferenced coordinates for the upper and lower corners. But in this case, the spatial extent of the raster is the enclosing geometry in its model space coordinate system. In Oracle GeoRaster, this is not necessarily in this way. Better idea? Worse idea? Honestly, I don’t know. An expert may surely provide and smarter point than mine. Reference: Oracle Georaster Developer’s Guide, section 3.6

Making buffers around the caribou points

Next step is to make 1km buffers around the caribou points, and reproject our buffers to WGS84, as our raster coverage. We’ll later use the buffers as sampling window for calculating statistics over an area of the GeoRaster data using SDO_GEOR.generateStatistics. We use a sampling window with Oracle GeoRaster because we can’t intersect vector and raster data. The closer operation is the one performed by SDO_GEOR.generateStatistics, or SDO_GEOR.subset. This is, select a window to clip an area of the raster and get statistics or raster data over this area.

Problem: The cariboupoints table doesn’t have a valid SRID. I forgot to specify it when dumping the table from PostGIS. We can set the SRID now:

Cariboupoints table update

We have to update the metadata too:

Cariboupoints metadata update

Problem: As we can read in SDO_GEOR.generateStatistics documentation, the sampling window must be rectangular, so, we can’t create round buffers around the cariboupoints. My solution: use the MBR of the buffers as sampling window.

Round buffers with MBR

This is the SQL code for creating the table with rectangular buffers around cariboupoints elements:

create table cariboupoint_buffers_wgs AS
select t.id, sdo_geom.sdo_mbr(sdo_geom.sdo_buffer(sdo_cs.transform(t.geom,
4326), 1000, 1)) geom
from cariboupoints t;

 
Now, we create an index over the polygons:

DELETE FROM user_sdo_geom_metadata WHERE table_name = 'cariboupoint_buffers_wgs' AND 
column_name = 'geom';
INSERT INTO user_sdo_geom_metadata VALUES ('cariboupoints_buffers_wgs', 'geom', 
SDO_DIM_ARRAY(SDO_DIM_ELEMENT('X', -180, 180, .00000005),
SDO_DIM_ELEMENT('Y', -90, 90, .00000005)), 4326);
DROP INDEX spain_images_sidx;
CREATE INDEX cariboupoints_buffers_wgs_gidx ON cariboupoints_buffers_wgs(geom) 
INDEXTYPE IS mdsys.spatial_index;

 
And in PostGIS?: As in Oracle Spatial, when you create a new table, you can create a GiST index over the spatial column (or a R-Tree index, but GiST indexes are recommended). However, I think the process in PostGIS is simpler: you only have to create the table and then the index.  In Oracle, you first have to update the metadata view, as seen. And you need to know not only the srid of your spatial data (easy to get) but the dimensions limits, and provide a tolerance, one concept you doesn’t need to necesarily know in Oracle’s context.

Let’s continue. Now, we show in OpenJump 2 layers:

  • One layer with the extent of our rasters (we can’t polygonize them with Oracle GeoRaster, like in PostGIS WKT Raster). SRID 4326.
  • One layer with 1 KM rectangular buffers around cariboupoints. srid 4326.

Rectangular Buffers and spatial extent of the raster data

Closer screenshot

Extents with intersections (closer)

Intersecting the caribou buffers with the elevation rasters

Big Problem: As said, you can’t intersect vector raster data with Oracle Database. Oracle Spatial has some spatial operators, and a geometry package, and you can perform some spatial operations. But Oracle GeoRaster doesn’t have this capability.

The solution adopted was to:

  1. Calculate what are the buffers that intersects the spatialExtent of the GeoRaster objects.
  2. Using these buffers as sampling windows, generate statistics for each pair buffer-georaster object, and store these statistics in a table
  3. Using the statistics generated, calculate the weighted mean elevation of the raster areas delimited by the buffers, and store the results in another table.

We can see these 3 steps in the next piece of PL/SQL code:

declare
cellCoordinate mdsys.sdo_geometry;
ret varchar2(256);
generated_statistics sdo_number_array;
gr sdo_georaster;
begin
-- create statistics table
create table srtm_caribou_inter_statistics (
id number, -- buffer id
rid number,
statistics sdo_number_array
);
-- Intersects spatial extent of the rasters with the buffers
for intersection in
(select t.id, t.geom, r.image_id as rid from spain_images r,
cariboupoint_buffers_wgs t
where sdo_geom.sdo_intersection(r.image.spatialExtent, t.geom, 0.005)
is not null)
loop
-- Get georaster image
select image into gr from spain_images where image_id
= intersection.rid for update;
-- First, we need geom coordinates in raster space coordinates
sdo_geor.getCellCoordinate(gr, 0, intersection.geom,
cellCoordinate);
-- Generate statistics for a given georaster object
ret := sdo_geor.generateStatistics(gr,
'samplingFactor=1', cellCoordinate, 'FALSE', '1-1', 'TRUE', NULL,
'TRUE');
-- update object
update spain_images r set r.image = gr where
r.image_id = intersection.rid;
-- commit changes
commit;
-- get generated statistics
generated_statistics := sdo_geor.getStatistics(gr, 1);
-- insert them in statistics table
insert into srtm_caribou_inter_statistics(id, rid, statistics) values
(intersection.id, intersection.rid, generated_statistics);
end loop;
commit;
end;
/

 
The code can be improved, as Jeffrey Xie suggested in this post, by using the result of the intersection operation as sampling window.  Anyway, this code runs very fasts. Less than 2 minutes. Only 212 buffers of the 814 intersects with the raster extents. Here, we can see a screenshot of the raster data and the intersecting buffers only (212). We can compare it with the previous OpenJUMP screenshot, with all the points (814).

Extents with intersections

And WKT Raster?: In WKT Raster we can create round buffers, and really intersects these polygons with the raster data. But the intersection is much slower, for 2 reasons:

  1. To really intersects vector and raster data (NOT vector with raster MBRs), we have to polygonize the raster, and then intersects these polygons with the vector data. This is a really heavy operation. In Oracle, you only check intersection betweem MBR, and only 212 of the 814 buffers intersect with the raster
  2. The sampling windows used in Oracle GeoRaster for intersection are rectangular. The buffers intersected with the raster data in WKT Raster are not rectangular.

Summarizing the elevation values for each buffer and exporting the table to CSV

As Pierre did, our last step is to compute the weighted mean elevation raster.  We’ll do in four steps:

  • Create a table to store the mean elevation data. The SQL code is:
    create table means_table(id number, rid number, mean number)
  • Fill the table. We have a problem here, because the varray field (the statistics obtained by SDO_GEOR.generateStatistics) must be cast to a nested table to access its fields, like we can read here. The PL/SQL code to do this is:
declare
st_array sdo_number_array;
sql_stmt varchar(256);
begin
for c1 in (select * from srtm_caribou_inter_statistics) loop
for i in c1.stats_array.FIRST..c1.stats_array.LAST loop¡
if i = 3 then

sql_stmt := 'insert into means_table values (:1, :2, :3)';
execute immediate sql_stmt using c1.id, c1.rid, c1.stats_array(i);
end if;
end loop;
end loop;
commit;
end;

 

  • Generate mean elevation data, by this query:
    select m.id, m.mean*sdo_geom.sdo_area(sdo_cs.transform(cpb.geom,
    32198), 1)  / sdo_geom.sdo_area(sdo_cs.transform(cpb.geom, 32198), 1)
    as meanelev from means_table m, cariboupoint_buffers_wgs cpb where cpb.idm.id;

     

    • Export the data to a CSV file. We need more PL/SQL for this. In this page there is an excellent method to dump query results in a csv file.

      Conclussions

      With Oracle GeoRaster you can do a basic raster/vector overlay analysis: compute pixel value statistics on areas delimited by vector polygons. But, in my opinion, it’s a hard task, because of non-intuitive tools and operations, and the fact Oracle GeoRaster wasn’t thought for spatial analysis, but mainly for raster data storage. And even using it for raster data storage, I think it’s a bit difficult. On the other hand, with PostGIS WKT Raster, you can perform this kind of operations in an easier and more intuitive way, at least for a non-experienced user, like me.

      Next post: comparing the ideal raster-on-database support with both extensions: Oracle GeoRaster and PostGIS WKT Raster

      Comparing Oracle GeoRaster with PostGIS WKT Raster (I)

      July 19, 2010

      UPDATE 2010-08-08: I’ve rewritten this post, with more interesting stuff.

      Pierre Racine wrote an excellent tutorial on PostGIS WKT Raster working with raster and vector data. As I’m involved in the development of WKT Raster extension, I’d like to know about similar solutions. The closest one is Oracle GeoRaster [PDF], in the sense is a raster-on-database solution.

      I thought: “Could I repit the Pierre’s tutorial with Oracle GeoRaster? would it be easier, harder?”. Now, my impressions:

      • My machine: Intel Core 2 Duo 3 GHz, 3 GB RAM
      • My Operating System: Windows XP SP3
      • My Oracle version: Oracle Database 11g Release 1 Enterprise Edition (11.1.0.6.0)

      PREVIOUS STEPS

      First step: Install Oracle Database

      Problem: After installing the software in my machine, it takes a loooooong time to start. Ok, Oracle Database Enterprise is a really powerful software, and is thought for lots of things PostGIS WKT Raster isn’t. It’s only for people simply looking for a raster-in-database solution, this one could be as using a sledgehammer to crack a nut. Only first impression.

      And WKT Raster?: The installation of PostgreSQL + PostGIS + WKT Raster is much lighter than the installation of Oracle Database 11g Enterprise. But Oracle Database has more components. We can’t compare both things. The point is, if you only need a raster-on-database solution, WKT Raster is a lighter one.

      Now, we’re going to create a new user. We execute a command shell and open SQL*Plus as sysdba (using  sys user):

      sqlplus sys/password as sysdba

      Once identified, execute:

      create user jorge identified by jorgejorgedefault tablespace users quota unlimited on
      users temporary tablespace temp; grant create session to jorge;

      When we create a new user, a new schema is automatically created, to store all the stuff generated by the new user. In this case, the schema is created in the users tablespace. We need to ensure the new user has the create session privilege granted, to start session in the Oracle backend.

      Next step, grant the new user enough privileges to create and manage tables. The easy way:

      grant all privileges to jorge

      Now, let’s see how to load shapefiles into Oracle Database.

      LOAD SHAPEFILES INTO ORACLE DATABASE USING ORACLE SPATIAL

      Problem: To import shapefiles into an Oracle Database, you must convert them to SDO format using Oracle Shapefile Converter.

      And PostGIS?: The ESRI Shapefile Format [PDF] is the format accepted by the PostGIS loader. So, you have to transform your data to this format if you want to use it, just like Oracle’s case. In both cases, instead of official loader you can use ogr2ogr to directly load the data into the database.

      Download Oracle Shapefile Converter here. The last version is from 2005, and the examples of the README file were made using Oracle 9, but still work. I converted a shapefile of the Ibiza island (in Spain) to SDO format. Here, the line:

      Oracle Shapefile Loader

      The shapefile is in UTM system, SRID 23030. You can find the x and y bounds here. As result, shp2sdo generates 2 files:

      • SQL file: With the Ibiza’s table and the metadata
      • CTL file: With the data and control sentences for SQL*Loader

      Time to load the SDO file in database. We use SQL*Loader:

      SQL*Plus loading file

      SQL*LDR loading control file

      We now have a new table created: IBIZA_COVERAGE:

      Shapefile loaded in Oracle

      Now, let’s work with raster data

      LOAD RASTER DATA INTO ORACLE DATABASE USING ORACLE GEORASTER

      We’ll need raster data. I’ve downloaded GeoTIFF files from Spain and Canary Island, by selecting them in the map. 6 files: srtm_35_04.tif, srtm_35_05.tif, srtm_36_04.tif, srtm_36_05.tif, srtm_33_07.tif, srtm_34_07.tif.

      Problem: The GeoRaster loader does not support source raster files in BSQ interleaving, and it might raise memory errors if the files are too big (how much is “too big”?). To avoid this problems, the Oracle GeoRaster developers guide recommends reformat and reblock the images using gdal_translate:

      gdal_translate -of GTiff -co “TFW=YES” -co “INTERLEAVE=PIXEL” -co “TILED=YES”
      image.tif image_new.tif

      This line creates a stripped image_new.tif file based in the original image.tif, but with pixel interleaving. After applying this filter to the six images,  they’re ready for loading in Oracle GeoRaster.

      And WKT Raster? The storage used for bands in WKT Raster is BSQ, but you don’t have to take care of the band interleaving of the raster you want to load. As the loader is based in GDAL, GDAL takes care of this.

      Problem: We need to first create the tables to store the raster data. These data are stored as a relationship between two tables. One table stores the metadata, and another one stores the raster data itself. And you need two tables for each raster you want to store. The software doesn’t provide an automatic tool to perform the create-needed-tables-and-load-data operation in one step.

      And WKT Raster? The loader creates the needed tables, the indexes (if specified in command line options) and store the data in them. To be honest, the loader generates a SQL file that you can load on Postgres, and makes all the stuff. Anyway, is easier.

      Let’s create the needed tables to store the downloaded GeoTIFF images of Spain:

      CREATE TABLE spain_images (image_id NUMBER PRIMARY KEY,
      image_description VARCHAR2(50), image SDO_GEORASTER);
      CREATE TABLE spain_images_rdt OF SDO_RASTER
      (PRIMARY KEY (rasterID, pyramidLevel, bandBlockNumber, rowBlockNumber, columnBlockNumber))
      TABLESPACE users LOB(rasterBlock) STORE AS SECUREFILE lobseg (NOCACHE);

      Two important things to remark here:

      Once tables have been created, we have three options for loading the raster data:

      • Use the PL/SQL loader
      • Use the Java loader
      • Use the Java-based GeoRaster visualizator.

      Problem: The two last options require to download and install the Oracle Database 11g examples. About 500MB to download. Again, this using a sledgehammer to crack a nut impression.

      And WKT Raster? When you install the WKT Raster extension on a PostGIS-enabled database, you have all you need. Even some test code. Again, Oracle Database Enterprise is much bigger. So, you can’t compare both solutions in this sense.

      Anoter important issue: One important advantage of WKT Raster over Oracle GeoRaster is the georeferencing approach. In case of non-rectangular raster coverages, this is particularly clear. Better to see with this two images:

      Oracle GeoRaster approach

      PostGIS WKT Raster approach

      That basically means that Oracle GeoRaster requires one raster data table for each raster data file loaded, if these raster files are from an irregular raster coverage. PostGIS WKT Raster allows only one table for all kinds of raster coverages. Regular or not. You can get further information in the WKT Raster FAQ

      Load raster data using PL/SQL loader

      The first step is to grant permissions to our user and to MDSYS for reading the file we want to load, using a Java-based program.

      call dbms_java.grant_permission(‘JORGE’, ‘SYS:java.io.FilePermission’,
      ‘C:\orcl_tut\image_new.tif’, ‘read’);
      call dbms_java.grant_permission(‘MDSYS’, ‘SYS:java.io.FilePermission’,
      ‘C:\orcl_tut\image_new.tif’, ‘read’);

      We repit the same operation with the rest of the files. Then, we can load the images, using a little PL/SQL code.  Here, a real executed piece of code:

      DECLARE
      geor SDO_GEORASTER;
      BEGIN
      -- Initialize an empty GeoRaster object into which the external image
      -- is to be imported.
      INSERT INTO spain_images
      values( 1, 'Spain_TIFF_1', sdo_geor.init('spain_images_rdt') );
      -- Import the TIFF image.
      SELECT image INTO geor FROM spain_images
      WHERE image_id = 1 FOR UPDATE;
      sdo_geor.importFrom(geor, 'blocksize=(256,256)', 'TIFF', 'file', 'C:\orcl_tut\srtm_35_04_new.tif');
      UPDATE spain_images SET image = geor WHERE image_id = 1;
      END;

      You have to repit the same insert-select-update cycle for all the images you want to load. Is striking you first have to insert an empty value in the data table, and then call a PL/SQL procedure to store the data in it.

      Problem: There are only a few image formats accepted. Only TIFF, GIF, PNG, BMP and GeoTIFF. JPEG is not supported by this method, but you can use Java loader for importing JPEG images.

      And WKT Raster? The raster formats allowed for loading are the ones allowed for your version of GDAL. You can check the GDAL accepted raster formats here. Anyway, for sure you can use more than six formats with a default installation of GDAL.

      Now, you have all your images loaded into database. We can use the validategeoraster procedure to ensure the information is correctly stored:

      GeoRaster data correctly loaded

      Load raster data using Java loader

      If you want to use the Java loader, you need to download the database examples from Oracle site. Take care of downloading the file that matchs your database version. Once installed, you’ll see a new md/demo folder inside your %ORACLE_HOME% directory ($ORACLE_HOME in Linux. This variable is set during Oracle Database installation). In that folder, there’s a lot of examples of all the technologies present in Oracle Database Enterprise. We are interested in the Georaster folder.

      GeoRaster examples folders

      Inside this Georaster folder, there are two more folders:

      • plsql: Some pieces of PL/SQL code to create GeoRaster tables, delete them, modify them, insert values, etc. Simple examples, resembling the code above.
      • java: here you hava the image loader, the image exporter and the image viewer. There’s a README file, explaining how to use them.

      To execute the java code, we first have to add some jar files to the CLASSPATH enviroment variable:

      CLASSPATH=%CLASSPATH%;%ORACLE_HOME%/ord/jlib/jai_codec.jar;%ORACLE_HOME%/ord/jlib/jai_core.jar;
      %ORACLE_HOME%/jdbc/lib/ojdbc5.jar;%ORACLE_HOME%/rdbms/jlib/xdb.jar;%ORACLE_HOME%/lib/xmlparserv2.jar;
      %ORACLE_HOME%/lib/xmlcomp.jar;%ORACLE_HOME%/lib/xschema.jar;%ORACLE_HOME%/jlib/jewt4.jar;
      %ORACLE_HOME%/md/jlib/sdoapi.jar;%ORACLE_HOME%/md/jlib/sdoutl.jar;
      %ORACLE_HOME%/md/demo/georaster/java/georaster_tools.jar

      Additionally, we’ll add xtiff-jai.jar and geotiff-jai.jar, because we want to load and export GeoTIFF images. We download these files, put them into a folder, and add the path to that folder to the CLASSPATH enviroment variable.

      Problem: In the README file, the JDBC driver is referred as ojdbc14.jar, and old version. But the driver available with this installation of Oracle is the newer ojdbc5.jar. An error caused by copy&paste of an older version of README, I guess.

      Now, we can execute the loader with one of the files:

      java oracle.spatial.georaster.tools.GeoRasterLoader localhost orcl 1521 jorge mypassword 
      thin 32 T images image "blocking=true,blockSize=(256,256),geotiff=true,srid=4326"
      "srtm_34_05_new.tif,1,rdt1,srtm_34_05_new.tfw,4326"

      Remember we blocked the image with gdal_translate. And the SRID of the files can be fetched with gdalinfo. The image formats accepted are the same than with PL/SQL loader plus JPEG and JP2.

      Problem: This happened when loading with PL/SQL, but in this case, I find it even less intuitive. If you want to add raster data, you first have to create an empty entry in data table, and then provide the name of this table and the id of the new record to the Java loader (in the example, the raster data table is rdt1, and the id is 1).

      And WKT Raster? As said, the whole process of create needed tables and insert data, can be performed in one step with WKT Raster. Much simpler.

      So, before calling the loader, you’ll have to execute an INSERT sentence like this:

      INSERT INTO images
      values( 1, sdo_geor.init('image', 1) );

      The important thing here is you can provide an id to sdo_geor.init procedure, and this will be the id you’ll have to use with the new image when calling the loader. If you don’t provide an id number as second argument of sdo_geor.init procedure, the system will choose the first free one. Write it down, to use it when calling the loader. Not very clear, in my opinion.

      Minor problem: There is a parameter, the eighth one, that has always the same value: T (interpreted as true, according with documentation). I suppose it’ll be used in future releases, or it will be deleted.

      Load raster data using GeoRaster visualizator

      The GeoRaster visualizator is a program written in Java that allows to load, view and export GeoRaster data. You can execute the visualizator by running the script startGeoRasterViewer.bat (or startGeoRasterViewer.sh if you’re in a UNIX enviroment).

      Problem: The Windows script (startGeoRasterViewer.bat) has 1 error. I had to fix it manually: At the beggining of the script, there’s a goto sentence that drives the code to an unexsistent tag (gotHome). The correct tag is moreCheck2.

      Now, we can execute the script file and load the GeoRaster viewer. As we can see, the images can be loaded from a disk file or retrieved from database.

      GeoRaster viewer

      We are interesting in fetching images from database, so, we provide our credentials. The viewer loads the GeoRaster data found in our schema. Here, a screenshot

      GeoRaster viewer showing tables

      In the menu bar, the third button from the right is the loading button. If we push it, a dialog will ask us for needed data, as host name, port number, database name, user name, password (yes, again). It will ask for the table and column name where we want to load our data. And for the raster id and the raster data table…

      GeoRaster viewer importing

      Problem: As happened when using java loader, you first have to create a new entry in raster data table by calling sdo_geor.init PL/SQL procedure, and be careful with raster id. Not very intuitive.

      VIEW RASTER DATA FROM ORACLE GEORASTER

      To view the raster data loaded into the database, you can use GeoRaster viewer. Of course you can use any other tools that allow connection with Oracle, but I want to know which tools are provided during a common installation of Oracle Database.

      Problem: There are two things I don’t like about GeoRaster viewer:

      • In README file states that Oracle GeoRaster provides, among others, the following feature: “supporting box selection on the GeoRaster object in the image panel, and allowing user to export  the selected window to the image file in the local file system”.  Well, I can’t find the way of doing it. It’s possible, when you export data, to specify init and end row and column, but not to directly select parts of the image in the viewer.
      • There’s a None option in the menu Image that seems to be a kind of  undo option, but I’m not sure.  A bit confusing option. See the screenshot:

      GeoRaster viewer "None" option

      Apart from this, the viewer is an acceptably useful tool for doing some basic operations on rasters

      And WKT Raster? WKT Raster doesn’t have a GUI, or a visualizator distributed packed with the code. Using GDAL WKT Raster driver (basic version right now, read only. It’s in my TODO list) you can connect WKT Raster with any program that uses GDAL for raster reading, but honestly, I haven’t tested it. It’s planned to work in this point.

      EXPORT RASTER DATA FROM ORACLE GEORASTER

      Again, we have three ways of exporting data from Oracle GeoRaster:

      • Using PL/SQL exporter
      • Using Java-based exporter
      • Using Java GeoRaster viewer

      Export raster data using PL/SQL exporter

      As in the importing option, the first step is, as sysdba, grant permissions to our user, and to MDSYS. Write permissions, this time:

      call dbms_java.grant_permission(‘JORGE’, ‘SYS:java.io.FilePermission’,
      ‘C:\orcl_tut\export\fichero.tif’, ‘write’);
      call dbms_java.grant_permission(‘MDSYS’, ‘SYS:java.io.FilePermission’,
      ‘C:\orcl_tut\export\fichero.tif’, ‘write’);

      Now, a bit of real PL/SQL again:

      DECLARE
      geor SDO_GEORASTER;
      BEGIN

      SELECT image INTO geor FROM spain_images WHERE image_id = 1;

      -- Export the whole GeoRaster object into a GeoTIFF file.
      sdo_geor.exportTo(geor,NULL, 'GeoTIFF', 'file',
      'C:\orcl_tut\export\image_exported_by_plsql');
      END;

      Problem: Formats accepted for image exporting using this method are only TIFF, BMP, GeoTiff or PNG. If you want JPEG or GIF, you have to use the Java exporter.

      And WKT Raster? As for reading, the formats allowed for writting are GDAL-dependent. You can use gdal_translate for exporting data, but as I’ve said, the GDAL WKT Raster driver is still a basic one. Needs more work and testing. Apart from this, there are JPEG and TIFF exporters planned, internally builded, not GDAL dependent.

      Executing this code, we get…

      GeoRaster exporting error with PL/SQL

      … yes, a memory error.

      Problem: In the Oracle GeoRaster developers guide, we read that “The maximum amount of GeoRaster data that can be exported in a single operation is 67 megabytes”. Our raster data are greater than this (files of 75MB). But the most strange thing is during the import operation we didn’t have that problem.

      And WKT Raster? The export size limits are determined by the file format itself, and the operating system/file system capabilities. This is, again, a GDAL issue. WKT Raster itself doesn’t impose a limitation. Further information here.

      Ok, we can export only a fragment of the data. Using the viewer, we get init and end row and column, to see a significative part of our files. Let’s try it again, this time with the cropArea option:

      DECLARE
      geor SDO_GEORASTER;
      BEGIN

      SELECT image INTO geor FROM spain_images WHERE image_id = 1;

      -- Export the whole GeoRaster object into a GeoTIFF file.
      sdo_geor.exportTo(geor, ’cropArea=(2024,951,2831,1663)’, 'GeoTIFF', 'file',
      'C:\orcl_tut\export\image_exported_by_plsql’);
      END;

      The file name without extension. Now the export operation finishes. Here, the result:

      GeoRaster image exported with PL/SQL

      I personally think  this is a severe limitation. Are 67 MB too much megabytes for Oracle GeoRaster?, why?

      Export raster data using Java-based exporter

      The Java exporter works in a similar way than the Java importer.

      Problem: We have to provide some parameters, like raster data table and raster id, that could be not very intuitive. We have to remember the raster data table name and the raster id we used when loading the data if we want to export the data loaded from the original files.

      And WKT Raster? As said, for data exporting, you use gdal_translate. Not well tested yet. But the needed options are clearly explained here

      Minor problem: In the README file, by the way, there is a minor typo when explaining the exporter calling method. There’s a repeated parameter in parameter list. Not very important, but it should be fixed in next releases.

      Then, the instruction to export one of the raster previously imported is:

      java oracle.spatial.georaster.tools.GeoRasterExporter localhost 1521 user password thin
      32 T images image "1,rdt1" "GEOTIFF" "export_file.tif"

      This time, the file name with extension, not like in the PL/SQL example. The formats accepted are the same than with the Java importer: TIFF, GeoTIFF, JPEG, BMP, GIF, PNG and JP2. After executing this line, we get…

      GeoRaster Java exporter memory error

      … yes, a memory error again.

      Problem: The raster is too big to be exported. The maximum size is, again, 67 MB. And we didn’t have this problem while loading with the Java importer.

      But this time, we have another solution, to get the whole raster exported, instead of using the cropArea method: the Java VM -Xmx option.

      GeoRaster Java exporter ok with more heap memory

      Then, if we increase the maximum heap size for our Java VM, we can get the whole raster data. I think it shouldn’t be necessary. I didn’t find these limitations working with GDAL and PostGIS WKT Raster.

      Export raster data using Java GeoRaster viewer

      Finally, we can export our data by using the Java GeoRaster Viewer. In the screenshot, the menu options available for doing this operation are marked in red

      GeorRaster viewer exporting

      Then, we provide some basic parameters (see screenshot)

      GeoRaster viewer exporting parameters

      Problem: If we choose GeoTIFF as format, we don’t need a World File. The program should inform about this, because while importing with the GeoRaster viewer, we have such option.

      Ok, we accept the options and then…

      Big problem: Yes, memory problem again. But the thing is the program doesn’t inform us about this problem. Actually, says the data were exported correctly! And the file has not been exported. I think this is an important problem, and must be fixed. See screenshot

      GeoRaster viewer exporting error

      In case of selecting any different format from GeoTIFF, and tell the viewer to generate a World File, it’s correctly generated, but the image file, as we’ve seen, doesn’t. The parameters of the World File are generated using scientific notation, but are correct.

      Of course, we can execute the GeoRaster viewer using -Xmx parameter, or export only an area of the raster, but I think this is not a good solution.

      CONCLUSSIONS:

      With Oracle GeoRaster, you can have a basic raster-in-database support, but I’m surprised to find bugs, important limitations and non-intuitive tools in an extension of a commercial software like this. Problems that, in most cases, WKT Raster has solved, or never suffered from them.

      The Oracle GeoRaster project seems to be something left since 2007. Next days, I’m going to delve into the tool, comparing its functionality with PostGIS WKT Raster, following the Pierre’s tutorial

      Calendar with upcoming events

      June 19, 2010

      I’ve created a public Google Calendar with important upcoming events related with Open Source Geospatial world. The information is gathered from http://www.osgeo.org/events. I’ll keep the calendar updated, so, you can subscribe by using the links on the right column, at section “Upcoming events”.

      UPDATE 2010-06-29:  The events have been added to the geomatic calendar created by Jorge Sanz.  Now, we work together in the same calendar.  So, the subscription links are from this calendar.

      Time to spread the word…

      June 19, 2010

      Ok, this is a milestone.

      The functions for Objective 0.1.6e are implemented and being tested (thanks to Regina Obe and Pierre Racine). I’m preparing presentations for España Virtual (my funding project) Workshop 2010 and, of course, for FOSS4G 2010.

      I had contact with some people from companies interested in WKT Raster, and I think this will be very positive for the project. And for me. There’s a lot of hard work to do, and a lot of new things to learn…

      GSoC 2010: not this time…

      May 1, 2010

      This summer I won’t participate on GSoC 2010. Anyway, I’m still working on GDAL WKT Raster driver and in WKT Raster extension. I’d like to update the blog more often, and I’ll probably do it from May…