http://www.libregis.org

So, you can update your markers if needed. I will keep this blog alive for some time.

• Web-viewer (ASP.NET)
• Web viewer (PHP)

• GISVM base: A VM with Ubuntu 10.10 Desktop and ready-to-execute scripts, that allow you to install the GIS software you need. The lighter version.
• GISVM desktop: This VM includes PostgreSQL, PostGIS, GeoServer, Mapserver, FWTools, QGIS, GRASS, gvSIG, uDIG, Kosmo and OpenJump installed. Based on Ubuntu 10.10 Desktop too.
• GISVM server: The sever version of the VM includes Linux, Apache, MySQL, Php,Tomcat,PostgreSQL, PostGIS, GeoServer, Mapserver, deegree and Geonetwork. Based on Ubuntu 10.10 server.

You can download the machines from here.

Do you want to collaborate in an Open Source project? Contact me by e-mail or leave a comment here. This summer could be a great summer of code!

PostGIS Raster was included as official part of PostGIS in October 2010. The old version, formerly named WKT Raster has been abandoned. So, if you want to use PostGIS Raster, with new functionalities and many bugs fixed, you’ll have to use the latest PostGIS development snapshot. The use of the old WKT Raster extension is strongly unrecommended, and there’s no support for it.

About GDAL, there was an important change in the driver on October 20st, 2010. A new version was committed, and this is the only version that works with PostGIS Raster. This new driver version was included in the release 1.8.0, currently the stable release. So, the use of an older release to connect with PostGIS Raster is also unrecommended.

With these facts in mind, I understand the Windows users who want to use PostGIS Raster and the GDAL driver are a bit unprotected. I’m not a Windows user, but I consider it my fault. For this reason, I’ve decided to create this howto. I’m going to install and configure PostGIS Raster and GDAL in Windows. And fortunately, you don’t need to compile anything. Let’s go for it:

Enviroment: Windows 7 32 bits

Step 1: Installing Python and Numpy

PostGIS Raster loader uses GDAL Python bindings and numpy. So, we’ll first install Python and numpy. Several options here:

• Python 2.5, unofficial numpy 1.5.1 (download numpy-1.5.1.win32-py2.5.‌exe from here)
• Python 2.6, unofficial numpy 1.5.1 (download numpy-1.5.1.win32-py2.6.‌exe from here)
• Python 2.7, unofficial numpy 1.5.1 (download numpy-1.5.1.win32-py2.7.‌exe from here)

I’m using:

• Python 2.6.6
• Numpy 1.5.1 (numpy-1.5.1.win32-py2.6.‌exe)

Step 2: Installing GDAL

I think the easiest way to have a complete GDAL installation (library + binaries + Python bindings) working on Windows is using the GDAL binaries mantained by Tamas Szekeres, here. Last GDAL versions from trunk are compiled daily with MSVC 2003, MSVC 2005 and MSVC 2008. Pick one. For example, the version compiled with MSVC 2003, here.

The  faster way now would be to download the generic installer for the GDAL core components, and choose the complete installation

GDAL 1.8.0 core installer for Windows

After the installation, you will need to add C:\Program Files\GDAL\ to the PATH enviroment var. And create this new var:

GDAL_DATA=C:\Program Files\GDAL\gdal-data

UPDATE 2011-10-07: About the GDAL Python bindings

After installing GDAL core components, you must install the Python bindings separately. You can use this installer (for Python 2.7 and GDAL 1.8.0), or grab one of the executables existent at http://www.lfd.uci.edu/~gohlke/pythonlibs/

Depending on your Python installation, you can choose GDAL-1.8.1.win32-py2.6.‌exe, GDAL-1.8.1.win32-py2.7.‌exe, GDAL-1.8.1.win32-py3.1.‌exe or GDAL-1.8.1.win32-py3.2.‌exe

UPDATE 2011-03-25: Alicia Duarte, from IDESF, told me she was to additionally set the next enviroment vars to make it work:

GDAL_DRIVER_PATH=C:\Program Files\GDAL\gdalplugins
PROJ_LIB=C:\Program Files\GDAL\projlib

WARNING: This directory (GDAL Python bindings location) may not exist in your GDAL installation. In my case, the bindings are in C:\Python27\Lib\site-packages\osgeo, and I didn’t need to define this variable.

PYTHONPATH=C:\Program Files\GDAL\python

UPDATE 2011-07-19:  I think I introduced some confusion in the comments. I apologize for that. Several people are having problems with GDAL Python bindings. I’ll  try to make it clearer:

• On one hand, there’s a sure thing: you need GDAL library + GDAL Python bindings installed in your system. There’s an easy way of doing it, by executing the generic installer: gdal-18-1310-core.msi. This installer includes the Python bindings, and they’re installed at C:\Program Files\GDAL\python directory by default. This should be enough.
• If you previously have a Python interpreter and a GDAL version installed on your system (check it from a console, executing gdalinfo –formats), you may want to simply install the proper GDAL Python bindings. Let’s assume your GDAL version is 1.8 (lower versions won’t work with PostGIS Raster), and your Python version is 2.7. Then, you simply need to install GDAL-1.8.0.win32-py2.7.msi. This installer will put GDAL Python bindings at C:\Python27\Lib\site-packages by default. This is a different location than above, but at the end, you should have a working version of GDAL Python bindings on your system.
• Finally,  the PostGIS Windows experimental binaries with PostGIS Raster support (step 4) are packed with their own version of GDAL library, used only by the raster2pgsql loader.

GDAL installed. Now, let’s go with step 3

Step 3: Installing PostgreSQL

I recommend Postgresql 8.4.x or higher. You can download the 1-click-installer from here, and install it in the common Windows’ way.

During the installation, you’ll be asked to install some nice additional stuff using the Application Stack Builder, like PostGIS 1.5.x or PostGIS 1.4.x. Don’t install anything. The PostGIS versions that can be installed doesn’t include PostGIS Raster. And you can’t install it separately, because, remember, now is part of PostGIS. We’re going to install last PostGIS version by hand. Don’t be afraid, it’ll be easy.

Step 4: Installing PostGIS

We’re going to use the PostGIS Windows experimental binaries. We need the binaries compiled for PostgreSQL 8.4, these ones

Once uncompressed, you can see this

Contents of binary package PostGIS 2.0 for Windows

Following the instructions of README.txt file, let’s edit makepostgisdb.bat to adapt the vars to our enviroment. We may want to change these vars:

set PGPORT=5432
set PGHOST=localhost
set PGUSER=postgres
set PGPASSWORD=password_of_postgres_user
set THEDB=template_postgis20
set PGBIN=C:\Program Files\PostgreSQL\8.4\bin\
set PGLIB=C:\Program Files\PostgreSQL\8.4\lib\

After that, we make a backup of our PostgreSQL bin and lib directories (PGBIN and PGLIB). Just in case.

We now execute makepostgisdb.bat. Take into account you may want to execute the script as administrator, in case your Windows user doesn’t have needed privileges (it was my case).

UPDATE 2011-06-08: For all Windows XP Professional SP2 users:

when you run makepostgisdb.bat from the binaries of POSTGIS SVN2.0, please rename the libiconv-2.dll in bin folder of Postgresql/8.4/bin folder to any other name. Now run the makepostgisdb.bat file. You will not get any error of “operation aborted …. transaction block” (Thanks to C Patnaik)

Once executed, we have PostGIS 2.0 + PostGIS Raster installed and configured. A new database called template_postgis20 is created, enabled with PostGIS and PostGIS Raster. Easy! Let’s test it with the next query:

select postgis_full_version(), postgis_raster_lib_build_date(), postgis_raster_lib_version()

And the result:

PostGIS version installed

Of course, you can create your own database using template_postgis20 as template. From the command line:

createdb -U postgres -T template_postgis20 your_database

Or from pgAdmin III:

Creating new database from template in pgAdmin III

If you want to use the SHP2PGSQL GUI, you must copy the postguisgui folder (inside bin folder in the downloaded PostGIS experimental binaries file) into pgAdmin III folder (mine is C:\Program Files\pgAdmin III\1.12) and add the next lines to plugins.ini file, inside pgAdmin III folder too:

;
;PostGIS shp2pgsql-gui (Windows):
;
Title=PostGIS Shapefile and DBF loader
Command="$$PGBINDIR\postgisgui\shp2pgsql-gui.exe" -h "$$HOSTNAME" -p $$PORT -U "$$USERNAME" -d "$$DATABASE" -W "$$PASSWORD"
Description=Open a PostGIS ESRI Shapefile or Plain dbf loader console to the current database.
KeyFile=$$PGBINDIR\postgisgui\shp2pgsql-gui.exe
Platform=windows
ServerType=postgresql
Database=Yes
SetPassword=Yes

After that, you can use the GUI inside pgAdmin III:

SHP2PGSQL GUI installed under pgAdmin III

Now you can load raster data into your database in 2 steps. In the first step, you will generate a SQL file. Open a command line and execute this:

python C:\Program Files\Postgresql\8.4\bin\raster2pgsql.py -r image.tif -t tablename -o image.sql

As result of this step, a SQL file named image.sql will be created in the directory where you executed the Python line above.

In the second step, you’ll need to load the SQL file in your_database, by using your favourite PostgreSQL client. For example, pgAdmin. If your use the psql command line client, the line to execute is:

psql -h localhost -U postgres -d your_database -f image.sql

If you want to translate your raster data from the database to image files at disk again, execute gdal_translate.

gdal_translate PG:"dbname=your_database user=postgres table=your_table mode=2" your_file.tif

Basically, this is all what you need to work with PostGIS Raster in Windows. Anyway, take into account the next additional notes:

Additional notes:

In step 2, I installed the whole GDAL library, including binaries and Python bindings. In step 4, I installed PostGIS 2.0 with all its pre-requisites. One of them is GDAL. The optimal way would be to use the existent GDAL library, but we’re working with pre-compiled binaries and DLLs. So, we have 2 copies of GDAL library:

• C:\Program Files\GDAL\gdal18.dll: Installed with generic installer for GDAL, in step 2. Used by gdal_translate, when reading data from database, and in any operation involving GDAL, except the raster loading with raster2pgsql.py script
• C:\Program Files\PostgreSQL\8.4\bin\libgdal.dll: Installed with PostGIS Windows experimental binaries, in step 4. Used by raster2pgsql.py script, to load raster data into PostgreSQL.

Both library files were compiled from GDAL 1.8.0 from trunk. But they will be slightly different, for sure. This is a suboptimal solution, of course, but it shouldn’t cause problems, because libgdal.dll is only used when loading raster data into the database. The rest of operations involving GDAL will be executed linking with gdal18.dll

If you want to translate the raster data to any image format different from TIFF, you can use the -of option in gdal_translate. For example:

gdal_translate -of "PNG" PG:"dbname=your_database user=your_user password=your_password table=your_table mode=2" your_file.png

To get available raster formats:

gdalinfo --formats

UPDATE 2011-06-11: I changed the gdal_translate line. I’m using the ‘postgres’ user and the ‘trust’ method to authenticate this user (so, no password is needed), but it’s only a simplification. In general, you should use a different user and a password-enabled authentication method, like md5. Thanks to C.Patnaik again, because his comment remainded me this issue

Conclussions:

It’s possible to work with the last version of PostGIS Raster and GDAL PostGIS Raster driver in Windows, but assuming that:

• You’re using experimental build. Not recommended for production enviroments.
• The previous versions of PostGIS Raster (WKT Raster) and GDAL PostGIS Raster driver (before GDAL 1.8.0) are discontinued. Don’t use them.
• It’s a suboptimal solution, because you install 2 copies of GDAL library. But I think this is the best available solution right now.

Do you have problems, something to add, modify or correct? Let me know it in the comments

• QGIS plugin for PostGIS Raster. Further information here.
• PostGIS Raster in gvSIG. Presentation in spanish, sorry. Official part of gvSIG from version 1.11.

Many thanks! These initiatives help to continue working

Apart from that, I’m working on PostGIS Raster MapAlgebra. First version before January (I hope…)

1. ./configure –with-raster
2. make
3. make rt-all
4. make install
5. make rt-install
6. createdb <your_database>
7. createlang plpgsql <your_database>
8. psql -f postgis/postgis.sql -d <your_database>
9. psql -f spatial_ref_sys.sql -d <your_database>
10. psql -f raster/rt_pg/rtpostgis.sql -d <your_database>

UPDATE 1 (2010-10-22): With the last release (r6095) it’s an easier process:

1. ./configure –with-raster
2. make
3. make install
4. createdb <your_database>
5. createlang plpgsql <your_database>
6. psql -U <your_user> -f postgis/postgis.sql -d <your_database>
7. psql -U <your user> -f spatial_ref_sys.sql -d <your_database>
8. psql -U <your user> -f raster/rt_pg/rtpostgis.sql -d <your_database>

In case your user requires a password, add -W as psql flag, and you’ll be prompted for the password. Further information in README.postgis and README.raster files.

And of course, I’m working on GDAL PostGIS Raster driver. I’ll add the new specifications ASAP.

UPDATE 2 (2010-10-21): New version of the driver committed to repository.  Now its name is postgisraster, instead the old one: wktraster. Further information in the link above.

Now, I’d like to reproduce the Pierre Racine’s tutorial, but using Oracle GeoRaster. Will it be possible? Let’s see it…

PREVIOUS STEPS

We’re going to use OpenJUMP, to see the results of our queries. So, we first install it. Easy task. But now, we need a plugin that allows OpenJUMP to retrieve geographic information from our Oracle Database. The plugin I used was JUMP DB Query Plugin.

The instructions are pretty clear, so, I only had to:

1. Unzip the plugin files into the OpenJUMP lib/ext folder
2. Download and unzip the correct Oracle JDBC driver for my Oracle Enterprise Database version (11.1.0.6.0) in the OpenJUMP lib/ext folder.
3. Run openjump.bat

And that’s all. Ready to start.

Loading shapefiles (caribou distribution shapefiles)

I used the cariboupoint table generated in PostGIS, just as described in the tutorial. Using pgsql2shp, I dumped the table data into a shapefile.

Dumping cariboupoints table

Then, using shp2sdo again, I converted my data to SDO format:

Transforming cariboupoints SHP to SDO format

Of course, I could create the data using a PL/SQL procedure, just like Pierre did in his tutorial. But the procedure was created only for providing a shapefile to work with, in case you don’t have access to real caribou distribution data. We can use the same file.

Visualizing the caribou point in OpenJUMP

Now, from OpenJUMP, run the query select geom from cariboupoints. The result, here:

Viewing cariboupoints table from Oracle in OpenJUMP

Loading raster data

As we showed in the previous post, we need to insert 4 GeoTIFF files, covering the whole area of Spain. Specifically, 4 SRTM 90m DEM files. There are 2 more files, covering the area of Canary Islands, but they are not continuous with the rest of the data. As Oracle GeoRaster uses a one-georeference-by-layer georeference schema instead of one-georeference-by-raster one, you can’t store non-continuous raster data in the same raster data table. For the purposes of this tutorial, we only need the 4 files covering Spain area.

So, as we saw in previous post, we first reformat and reblock GeoTIFF images, to ensure they don’t have BSQ interleaving:

gdal_translate -of GTiff -a_srs epsg:4326 -anodata -32768 -co “TFW=YES” -co “INTERLEAVE=PIXEL”
-co “TILED=YES” -co "BLOCKXSIZE=50" -co "BLOCKYSIZE=50" image.tif image_new.tif

 
Options used:

• -of Gtiff: The output format for the reformatted files. GeoTIFF again.
• -a_srs epsg:4326: This is the srid for the data. The same than input files.
• -anodata -32768: This is the NODATA value for bands. We can get this value using gdalinfo over the original files.
• -co “TFW=YES”: This forces gdal_translate to create an output tfw file, with georeference data.
• -co “INTERLEAVE=PIXEL”: Forces pixel interleaving in the output files, the only allowed interleaving schema.
• -co “TILED=yes”: Forces the creation of stripped TIFF files.
• -co “BLOCKXSIZE=50″, -co BLOCKYSIZE=50″: Dimensions for tiles. The same used in WKT Raster tutorial.

Further information about GeoTIFF driver for GDAL here. Now, we have to repit this with every single file we’re going to load. Once done, we can load the files in Oracle GeoRaster, using SDO_GEOR.importFrom procedure as follows:

DECLARE
geor SDO_GEORASTER;
BEGIN
-- Initialize an empty GeoRaster object into which the external image
-- is to be imported.
INSERT INTO spain_images
values( 1, 'Spain_TIFF_1', sdo_geor.init('spain_images_rdt') );
-- Import the TIFF image.
SELECT image INTO geor FROM spain_images
WHERE image_id = 1 FOR UPDATE;
sdo_geor.importFrom(geor, 'blocksize=(50,50) spatialExtent=TRUE', 'TIFF', 
'file', 'C:\orcl_tut\srtm_35_04_new.tif',
'WORLDFILE', 'FILE', 'C:\orcl_tut\srtm_35_04_new.tfw');
UPDATE spain_images SET image = geor WHERE image_id = 1;
END;

 
Three comments here:

• The option ‘blocksize=(50,50)‘ is not really needed. As we can read in the Oracle Spatial Developer’s guide, storage parameters section, “If you specify neither blocking nor blockSize, default values are derived from the source GeoRaster object: that is, if the original data is not blocked, the data in the output GeoRaster object is by default not blocked; and if the original data is blocked, the data in the output GeoRaster object is blocked with the same blocking scheme”. As long as we specified block size when reformatting GeoTIFF files, we wouldn’t need to use that option now.
• The option ‘spatialExtent=TRUE‘ will be necessary later. Anyway, if we omit this option, we can generate the spatial extent of the raster objects by SDO_GEOR.generateSpatialExtent procedure, once raster data are loaded. Further information on generating and setting spatial extents in the Oracle Georaster Developer’s Guide, section 3.6
• We provide, as additional arguments, the World File, with georeference information.

GeoRaster spatial extent ok

As you can see, the extent of the raster is represented as a polygon in SDO_GEOMETRY format. For me, looks complicated… If you want to know more about the meaning of all fields of the SDO_GEOMETRY object, you can check it here.

What is the reason of keeping a representation format so complicated? Ok, it could be my impression. After all, I’m new in this world. Anyway, I started a thread on PostGIS devel list, asking for this. One important topic that came up in conversation is Oracle has a topological storage model. But if you want to work with this model, you must use the SDO_TOPO package. And there’s probably no one tool able to work with this model, apart from Oracle itself, of course.

Take into account, in this particular case, the SDO_GEORASTER table is composed of 4 objects, because we had 4 GeoTIFF files. So, the extent is calculated for each object, not for the whole coverage. If we want only one object, we should:

• Merge the files before loading them.
• Use SDO_GEOR.mergeLayers

But, honestly, I didn’t try these solutions yet.

The last thing we have to do with raster data is… create and index. As we can read in Oracle GeoRaster developer’s guide, section 3.7, “the most important index you can create on a GeoRaster object is a spatial index on the spatial extent (footprint) geometry of the GeoRaster object”. And we need to take one more thing into account… According to the documentation (chapter 18), one of the prerequisites for creating an spatial index over a geometry column is “The USER_SDO_GEOM_METADATA view must contain an entry with the dimensions and coordinate boundary information for the table column to be spatially indexed”. And again, in the documentation (chapter 2, section 2.8): “Spatial users are responsible for populating these views. For each spatial column, you must insert an appropriate row into the USER_SDO_GEOM_METADATA view. Oracle Spatial ensures that the ALL_SDO_GEOM_METADATA view is also updated to reflect the rows that you insert into USER_SDO_GEOM_METADATA.” .

So, let’s do it:

DELETE FROM user_sdo_geom_metadata WHERE table_name = 'spain_images' AND 
column_name = 'IMAGE.SPATIALEXTENT';
INSERT INTO user_sdo_geom_metadata VALUES ('spain_images', 'IMAGE.SPATIALEXTENT', 
SDO_DIM_ARRAY(SDO_DIM_ELEMENT('X', -180, 180, .00000005),
SDO_DIM_ELEMENT('Y', -90, 90, .00000005)), 4326);
DROP INDEX spain_images_sidx;
CREATE INDEX spain_images_sidx ON spain_images(image.spatialExtent) INDEXTYPE IS mdsys.spatial_index;

Once done, our raster data is ready to work.

And WKT Raster?: All things we’ve done to load our raster data in Oracle GeoRaster and create an index, can be done in PostGIS WKT Raster by executing these two lines:

> gdal2wktraster.py" -r C:\orcl_tut\*.tif -t spain_images -s 4326 -k 50x50 -I 
-o C:\orcl_tut\srtm.sql
> psql -d tutorial01 -f C:\orcl_tut\srtm.sql

 
Of course, you need to install all the PostGIS stuff. You have an useful tutorial on how to do this here (in Portuguese, but I think it’s easy to follow. If you find it difficult, you can read WKT Raster documentation or ask in the PostGIS users list.

So far, I find too difficult the simple operation of loading GeoTIFF data in Oracle GeoRaster, comparing it with the same operation done with PostGIS WKT Raster. Of course you can always ask in Oracle Spatial forum (register required). The questions I’ve asked were quickly answered. Really great people in these forums.

Another important difference between both extensions are the indexes. Oracle GeoRaster creates spatial indexes over the footprint of raster data, and WKT Raster creates GiST indexes over the raster data itself. Just a newbie guessing: as far as I know, the operations you can do with raster data itself in Oracle GeoRaster are rather limited. Spatial operations, like intersections, can only be done with the MBR of the data, so, that’s the reason the indexes are created over the footprint. On the other hand, in PostGIS WKT Raster, you can do really heavy operations with raster data, like ST_DumpAsPolygons. Then, create an index over these raster data has more sense.

And if you want to interact with real raster data in Oracle GeoRaster? You can use the functionalities provided by SDO_GEOR package or directly attack the data blobs, using DBMS_LOB package. But this last option is a blind attack over raw binary data.

About the spatial extent of the raster data, in WKT Raster the spatial extent of a raster is calculated during the loading process, using GDAL provided raster dimensions, and calculating the georeferenced coordinates for the upper and lower corners. But in this case, the spatial extent of the raster is the enclosing geometry in its model space coordinate system. In Oracle GeoRaster, this is not necessarily in this way. Better idea? Worse idea? Honestly, I don’t know. An expert may surely provide and smarter point than mine. Reference: Oracle Georaster Developer’s Guide, section 3.6

Making buffers around the caribou points

Next step is to make 1km buffers around the caribou points, and reproject our buffers to WGS84, as our raster coverage. We’ll later use the buffers as sampling window for calculating statistics over an area of the GeoRaster data using SDO_GEOR.generateStatistics. We use a sampling window with Oracle GeoRaster because we can’t intersect vector and raster data. The closer operation is the one performed by SDO_GEOR.generateStatistics, or SDO_GEOR.subset. This is, select a window to clip an area of the raster and get statistics or raster data over this area.

Problem: The cariboupoints table doesn’t have a valid SRID. I forgot to specify it when dumping the table from PostGIS. We can set the SRID now:

Cariboupoints table update

We have to update the metadata too:

Cariboupoints metadata update

Problem: As we can read in SDO_GEOR.generateStatistics documentation, the sampling window must be rectangular, so, we can’t create round buffers around the cariboupoints. My solution: use the MBR of the buffers as sampling window.

Round buffers with MBR

This is the SQL code for creating the table with rectangular buffers around cariboupoints elements:

create table cariboupoint_buffers_wgs AS
select t.id, sdo_geom.sdo_mbr(sdo_geom.sdo_buffer(sdo_cs.transform(t.geom,
4326), 1000, 1)) geom
from cariboupoints t;

 
Now, we create an index over the polygons:

DELETE FROM user_sdo_geom_metadata WHERE table_name = 'cariboupoint_buffers_wgs' AND 
column_name = 'geom';
INSERT INTO user_sdo_geom_metadata VALUES ('cariboupoints_buffers_wgs', 'geom', 
SDO_DIM_ARRAY(SDO_DIM_ELEMENT('X', -180, 180, .00000005),
SDO_DIM_ELEMENT('Y', -90, 90, .00000005)), 4326);
DROP INDEX spain_images_sidx;
CREATE INDEX cariboupoints_buffers_wgs_gidx ON cariboupoints_buffers_wgs(geom) 
INDEXTYPE IS mdsys.spatial_index;

 
And in PostGIS?: As in Oracle Spatial, when you create a new table, you can create a GiST index over the spatial column (or a R-Tree index, but GiST indexes are recommended). However, I think the process in PostGIS is simpler: you only have to create the table and then the index.  In Oracle, you first have to update the metadata view, as seen. And you need to know not only the srid of your spatial data (easy to get) but the dimensions limits, and provide a tolerance, one concept you doesn’t need to necesarily know in Oracle’s context.

Let’s continue. Now, we show in OpenJump 2 layers:

• One layer with the extent of our rasters (we can’t polygonize them with Oracle GeoRaster, like in PostGIS WKT Raster). SRID 4326.
• One layer with 1 KM rectangular buffers around cariboupoints. srid 4326.

Rectangular Buffers and spatial extent of the raster data

Closer screenshot

Extents with intersections (closer)

Intersecting the caribou buffers with the elevation rasters

Big Problem: As said, you can’t intersect vector raster data with Oracle Database. Oracle Spatial has some spatial operators, and a geometry package, and you can perform some spatial operations. But Oracle GeoRaster doesn’t have this capability.

The solution adopted was to:

1. Calculate what are the buffers that intersects the spatialExtent of the GeoRaster objects.
2. Using these buffers as sampling windows, generate statistics for each pair buffer-georaster object, and store these statistics in a table
3. Using the statistics generated, calculate the weighted mean elevation of the raster areas delimited by the buffers, and store the results in another table.

We can see these 3 steps in the next piece of PL/SQL code:

declare
cellCoordinate mdsys.sdo_geometry;
ret varchar2(256);
generated_statistics sdo_number_array;
gr sdo_georaster;
begin

-- create statistics table
create table srtm_caribou_inter_statistics (
id number, -- buffer id
rid number,
statistics sdo_number_array
);

-- Intersects spatial extent of the rasters with the buffers
for intersection in
(select t.id, t.geom, r.image_id as rid from spain_images r,
cariboupoint_buffers_wgs t
where sdo_geom.sdo_intersection(r.image.spatialExtent, t.geom, 0.005)
is not null)

loop
-- Get georaster image
select image into gr from spain_images where image_id
= intersection.rid for update;

-- First, we need geom coordinates in raster space coordinates
sdo_geor.getCellCoordinate(gr, 0, intersection.geom,
cellCoordinate);

-- Generate statistics for a given georaster object
ret := sdo_geor.generateStatistics(gr,
'samplingFactor=1', cellCoordinate, 'FALSE', '1-1', 'TRUE', NULL,
'TRUE');

-- update object
update spain_images r set r.image = gr where
r.image_id = intersection.rid;

-- commit changes
commit;

-- get generated statistics
generated_statistics := sdo_geor.getStatistics(gr, 1);

-- insert them in statistics table
insert into srtm_caribou_inter_statistics(id, rid, statistics) values
(intersection.id, intersection.rid, generated_statistics);

end loop;
commit;
end;
/

 
The code can be improved, as Jeffrey Xie suggested in this post, by using the result of the intersection operation as sampling window.  Anyway, this code runs very fasts. Less than 2 minutes. Only 212 buffers of the 814 intersects with the raster extents. Here, we can see a screenshot of the raster data and the intersecting buffers only (212). We can compare it with the previous OpenJUMP screenshot, with all the points (814).

Extents with intersections

And WKT Raster?: In WKT Raster we can create round buffers, and really intersects these polygons with the raster data. But the intersection is much slower, for 2 reasons:

1. To really intersects vector and raster data (NOT vector with raster MBRs), we have to polygonize the raster, and then intersects these polygons with the vector data. This is a really heavy operation. In Oracle, you only check intersection betweem MBR, and only 212 of the 814 buffers intersect with the raster
2. The sampling windows used in Oracle GeoRaster for intersection are rectangular. The buffers intersected with the raster data in WKT Raster are not rectangular.

Summarizing the elevation values for each buffer and exporting the table to CSV

As Pierre did, our last step is to compute the weighted mean elevation raster.  We’ll do in four steps:

• Create a table to store the mean elevation data. The SQL code is:

  create table means_table(id number, rid number, mean number)

• Fill the table. We have a problem here, because the varray field (the statistics obtained by SDO_GEOR.generateStatistics) must be cast to a nested table to access its fields, like we can read here. The PL/SQL code to do this is:

declare

st_array sdo_number_array;

sql_stmt varchar(256);

begin

for c1 in (select * from srtm_caribou_inter_statistics) loop

for i in c1.stats_array.FIRST..c1.stats_array.LAST loop¡

if i = 3 then

sql_stmt := 'insert into means_table values (:1, :2, :3)';

execute immediate sql_stmt using c1.id, c1.rid, c1.stats_array(i);

end if;

end loop;

end loop;

commit;

end;

• Generate mean elevation data, by this query:

select m.id, m.mean*sdo_geom.sdo_area(sdo_cs.transform(cpb.geom,
32198), 1)  / sdo_geom.sdo_area(sdo_cs.transform(cpb.geom, 32198), 1)
as meanelev from means_table m, cariboupoint_buffers_wgs cpb where cpb.id = m.id;

• Export the data to a CSV file. We need more PL/SQL for this. In this page there is an excellent method to dump query results in a csv file.

Conclussions

With Oracle GeoRaster you can do a basic raster/vector overlay analysis: compute pixel value statistics on areas delimited by vector polygons. But, in my opinion, it’s a hard task, because of non-intuitive tools and operations, and the fact Oracle GeoRaster wasn’t thought for spatial analysis, but mainly for raster data storage. And even using it for raster data storage, I think it’s a bit difficult. On the other hand, with PostGIS WKT Raster, you can perform this kind of operations in an easier and more intuitive way, at least for a non-experienced user, like me.

Next post: comparing the ideal raster-on-database support with both extensions: Oracle GeoRaster and PostGIS WKT Raster

Pierre Racine wrote an excellent tutorial on PostGIS WKT Raster working with raster and vector data. As I’m involved in the development of WKT Raster extension, I’d like to know about similar solutions. The closest one is Oracle GeoRaster [PDF], in the sense is a raster-on-database solution.

I thought: “Could I repit the Pierre’s tutorial with Oracle GeoRaster? would it be easier, harder?”. Now, my impressions:

• My machine: Intel Core 2 Duo 3 GHz, 3 GB RAM
• My Operating System: Windows XP SP3
• My Oracle version: Oracle Database 11g Release 1 Enterprise Edition (11.1.0.6.0)

PREVIOUS STEPS

First step: Install Oracle Database

Problem: After installing the software in my machine, it takes a loooooong time to start. Ok, Oracle Database Enterprise is a really powerful software, and is thought for lots of things PostGIS WKT Raster isn’t. It’s only for people simply looking for a raster-in-database solution, this one could be as using a sledgehammer to crack a nut. Only first impression.

And WKT Raster?: The installation of PostgreSQL + PostGIS + WKT Raster is much lighter than the installation of Oracle Database 11g Enterprise. But Oracle Database has more components. We can’t compare both things. The point is, if you only need a raster-on-database solution, WKT Raster is a lighter one.

Now, we’re going to create a new user. We execute a command shell and open SQL*Plus as sysdba (using  sys user):

sqlplus sys/password as sysdba

Once identified, execute:

create user jorge identified by jorgejorgedefault tablespace users quota unlimited on

users temporary tablespace temp; grant create session to jorge;

When we create a new user, a new schema is automatically created, to store all the stuff generated by the new user. In this case, the schema is created in the users tablespace. We need to ensure the new user has the create session privilege granted, to start session in the Oracle backend.

Next step, grant the new user enough privileges to create and manage tables. The easy way:

grant all privileges to jorge

Now, let’s see how to load shapefiles into Oracle Database.

LOAD SHAPEFILES INTO ORACLE DATABASE USING ORACLE SPATIAL

Problem: To import shapefiles into an Oracle Database, you must convert them to SDO format using Oracle Shapefile Converter.

And PostGIS?: The ESRI Shapefile Format [PDF] is the format accepted by the PostGIS loader. So, you have to transform your data to this format if you want to use it, just like Oracle’s case. In both cases, instead of official loader you can use ogr2ogr to directly load the data into the database.

Download Oracle Shapefile Converter here. The last version is from 2005, and the examples of the README file were made using Oracle 9, but still work. I converted a shapefile of the Ibiza island (in Spain) to SDO format. Here, the line:

Oracle Shapefile Loader

The shapefile is in UTM system, SRID 23030. You can find the x and y bounds here. As result, shp2sdo generates 2 files:

Time to load the SDO file in database. We use SQL*Loader:

SQL*Plus loading file

SQL*LDR loading control file

We now have a new table created: IBIZA_COVERAGE:

Shapefile loaded in Oracle

Now, let’s work with raster data

LOAD RASTER DATA INTO ORACLE DATABASE USING ORACLE GEORASTER

We’ll need raster data. I’ve downloaded GeoTIFF files from Spain and Canary Island, by selecting them in the map. 6 files: srtm_35_04.tif, srtm_35_05.tif, srtm_36_04.tif, srtm_36_05.tif, srtm_33_07.tif, srtm_34_07.tif.

Problem: The GeoRaster loader does not support source raster files in BSQ interleaving, and it might raise memory errors if the files are too big (how much is “too big”?). To avoid this problems, the Oracle GeoRaster developers guide recommends reformat and reblock the images using gdal_translate:

gdal_translate -of GTiff -co “TFW=YES” -co “INTERLEAVE=PIXEL” -co “TILED=YES”
image.tif image_new.tif

This line creates a stripped image_new.tif file based in the original image.tif, but with pixel interleaving. After applying this filter to the six images,  they’re ready for loading in Oracle GeoRaster.

And WKT Raster? The storage used for bands in WKT Raster is BSQ, but you don’t have to take care of the band interleaving of the raster you want to load. As the loader is based in GDAL, GDAL takes care of this.

Problem: We need to first create the tables to store the raster data. These data are stored as a relationship between two tables. One table stores the metadata, and another one stores the raster data itself. And you need two tables for each raster you want to store. The software doesn’t provide an automatic tool to perform the create-needed-tables-and-load-data operation in one step.

And WKT Raster? The loader creates the needed tables, the indexes (if specified in command line options) and store the data in them. To be honest, the loader generates a SQL file that you can load on Postgres, and makes all the stuff. Anyway, is easier.

Let’s create the needed tables to store the downloaded GeoTIFF images of Spain:

CREATE TABLE spain_images (image_id NUMBER PRIMARY KEY,
image_description VARCHAR2(50), image SDO_GEORASTER);

CREATE TABLE spain_images_rdt OF SDO_RASTER
(PRIMARY KEY (rasterID, pyramidLevel, bandBlockNumber, rowBlockNumber, columnBlockNumber))
TABLESPACE users LOB(rasterBlock) STORE AS SECUREFILE lobseg (NOCACHE);

Two important things to remark here:

• We’re using LOB secure files storage schema. See Oracle Database SecureFiles and Large Objects Developer’s Guide and Oracle Database SQL Language Reference for further information on this topic.
• When we create the primary key for this table, a B-Tree index over data table is automatically created. PostGIS WKT Raster creates a GiST index for the same purpose.

Once tables have been created, we have three options for loading the raster data:

• Use the PL/SQL loader
• Use the Java loader
• Use the Java-based GeoRaster visualizator.

Problem: The two last options require to download and install the Oracle Database 11g examples. About 500MB to download. Again, this using a sledgehammer to crack a nut impression.

And WKT Raster? When you install the WKT Raster extension on a PostGIS-enabled database, you have all you need. Even some test code. Again, Oracle Database Enterprise is much bigger. So, you can’t compare both solutions in this sense.

Anoter important issue: One important advantage of WKT Raster over Oracle GeoRaster is the georeferencing approach. In case of non-rectangular raster coverages, this is particularly clear. Better to see with this two images:

Oracle GeoRaster approach

PostGIS WKT Raster approach

That basically means that Oracle GeoRaster requires one raster data table for each raster data file loaded, if these raster files are from an irregular raster coverage. PostGIS WKT Raster allows only one table for all kinds of raster coverages. Regular or not. You can get further information in the WKT Raster FAQ

Load raster data using PL/SQL loader

The first step is to grant permissions to our user and to MDSYS for reading the file we want to load, using a Java-based program.

call dbms_java.grant_permission(‘JORGE’, ‘SYS:java.io.FilePermission’,
‘C:\orcl_tut\image_new.tif’, ‘read’);

call dbms_java.grant_permission(‘MDSYS’, ‘SYS:java.io.FilePermission’,
‘C:\orcl_tut\image_new.tif’, ‘read’);

We repit the same operation with the rest of the files. Then, we can load the images, using a little PL/SQL code.  Here, a real executed piece of code:

DECLARE
geor SDO_GEORASTER;

BEGIN

-- Initialize an empty GeoRaster object into which the external image

-- is to be imported.

INSERT INTO spain_images

values( 1, 'Spain_TIFF_1', sdo_geor.init('spain_images_rdt') );

-- Import the TIFF image.

SELECT image INTO geor FROM spain_images

WHERE image_id = 1 FOR UPDATE;

sdo_geor.importFrom(geor, 'blocksize=(256,256)', 'TIFF', 'file', 'C:\orcl_tut\srtm_35_04_new.tif');

UPDATE spain_images SET image = geor WHERE image_id = 1;
END;

You have to repit the same insert-select-update cycle for all the images you want to load. Is striking you first have to insert an empty value in the data table, and then call a PL/SQL procedure to store the data in it.

Problem: There are only a few image formats accepted. Only TIFF, GIF, PNG, BMP and GeoTIFF. JPEG is not supported by this method, but you can use Java loader for importing JPEG images.

And WKT Raster? The raster formats allowed for loading are the ones allowed for your version of GDAL. You can check the GDAL accepted raster formats here. Anyway, for sure you can use more than six formats with a default installation of GDAL.

Now, you have all your images loaded into database. We can use the validategeoraster procedure to ensure the information is correctly stored:

GeoRaster data correctly loaded

Load raster data using Java loader

If you want to use the Java loader, you need to download the database examples from Oracle site. Take care of downloading the file that matchs your database version. Once installed, you’ll see a new md/demo folder inside your %ORACLE_HOME% directory ($ORACLE_HOME in Linux. This variable is set during Oracle Database installation). In that folder, there’s a lot of examples of all the technologies present in Oracle Database Enterprise. We are interested in the Georaster folder.

GeoRaster examples folders

Inside this Georaster folder, there are two more folders:

• plsql: Some pieces of PL/SQL code to create GeoRaster tables, delete them, modify them, insert values, etc. Simple examples, resembling the code above.
• java: here you hava the image loader, the image exporter and the image viewer. There’s a README file, explaining how to use them.

To execute the java code, we first have to add some jar files to the CLASSPATH enviroment variable:

CLASSPATH=%CLASSPATH%;%ORACLE_HOME%/ord/jlib/jai_codec.jar;%ORACLE_HOME%/ord/jlib/jai_core.jar;
%ORACLE_HOME%/jdbc/lib/ojdbc5.jar;%ORACLE_HOME%/rdbms/jlib/xdb.jar;%ORACLE_HOME%/lib/xmlparserv2.jar;
%ORACLE_HOME%/lib/xmlcomp.jar;%ORACLE_HOME%/lib/xschema.jar;%ORACLE_HOME%/jlib/jewt4.jar;
%ORACLE_HOME%/md/jlib/sdoapi.jar;%ORACLE_HOME%/md/jlib/sdoutl.jar;
%ORACLE_HOME%/md/demo/georaster/java/georaster_tools.jar

Additionally, we’ll add xtiff-jai.jar and geotiff-jai.jar, because we want to load and export GeoTIFF images. We download these files, put them into a folder, and add the path to that folder to the CLASSPATH enviroment variable.

Problem: In the README file, the JDBC driver is referred as ojdbc14.jar, and old version. But the driver available with this installation of Oracle is the newer ojdbc5.jar. An error caused by copy&paste of an older version of README, I guess.

Now, we can execute the loader with one of the files:

java oracle.spatial.georaster.tools.GeoRasterLoader localhost orcl 1521 jorge mypassword 
thin 32 T images image "blocking=true,blockSize=(256,256),geotiff=true,srid=4326"
"srtm_34_05_new.tif,1,rdt1,srtm_34_05_new.tfw,4326"

Remember we blocked the image with gdal_translate. And the SRID of the files can be fetched with gdalinfo. The image formats accepted are the same than with PL/SQL loader plus JPEG and JP2.

Problem: This happened when loading with PL/SQL, but in this case, I find it even less intuitive. If you want to add raster data, you first have to create an empty entry in data table, and then provide the name of this table and the id of the new record to the Java loader (in the example, the raster data table is rdt1, and the id is 1).

And WKT Raster? As said, the whole process of create needed tables and insert data, can be performed in one step with WKT Raster. Much simpler.

So, before calling the loader, you’ll have to execute an INSERT sentence like this:

INSERT INTO images
values( 1, sdo_geor.init('image', 1) );

The important thing here is you can provide an id to sdo_geor.init procedure, and this will be the id you’ll have to use with the new image when calling the loader. If you don’t provide an id number as second argument of sdo_geor.init procedure, the system will choose the first free one. Write it down, to use it when calling the loader. Not very clear, in my opinion.

Minor problem: There is a parameter, the eighth one, that has always the same value: T (interpreted as true, according with documentation). I suppose it’ll be used in future releases, or it will be deleted.

Load raster data using GeoRaster visualizator

The GeoRaster visualizator is a program written in Java that allows to load, view and export GeoRaster data. You can execute the visualizator by running the script startGeoRasterViewer.bat (or startGeoRasterViewer.sh if you’re in a UNIX enviroment).

Problem: The Windows script (startGeoRasterViewer.bat) has 1 error. I had to fix it manually: At the beggining of the script, there’s a goto sentence that drives the code to an unexsistent tag (gotHome). The correct tag is moreCheck2.

Now, we can execute the script file and load the GeoRaster viewer. As we can see, the images can be loaded from a disk file or retrieved from database.

GeoRaster viewer

We are interesting in fetching images from database, so, we provide our credentials. The viewer loads the GeoRaster data found in our schema. Here, a screenshot

GeoRaster viewer showing tables

In the menu bar, the third button from the right is the loading button. If we push it, a dialog will ask us for needed data, as host name, port number, database name, user name, password (yes, again). It will ask for the table and column name where we want to load our data. And for the raster id and the raster data table…

GeoRaster viewer importing

Problem: As happened when using java loader, you first have to create a new entry in raster data table by calling sdo_geor.init PL/SQL procedure, and be careful with raster id. Not very intuitive.

VIEW RASTER DATA FROM ORACLE GEORASTER

To view the raster data loaded into the database, you can use GeoRaster viewer. Of course you can use any other tools that allow connection with Oracle, but I want to know which tools are provided during a common installation of Oracle Database.

Problem: There are two things I don’t like about GeoRaster viewer:

• In README file states that Oracle GeoRaster provides, among others, the following feature: “supporting box selection on the GeoRaster object in the image panel, and allowing user to export  the selected window to the image file in the local file system”.  Well, I can’t find the way of doing it. It’s possible, when you export data, to specify init and end row and column, but not to directly select parts of the image in the viewer.
• There’s a None option in the menu Image that seems to be a kind of  undo option, but I’m not sure.  A bit confusing option. See the screenshot:

GeoRaster viewer "None" option

Apart from this, the viewer is an acceptably useful tool for doing some basic operations on rasters

And WKT Raster? WKT Raster doesn’t have a GUI, or a visualizator distributed packed with the code. Using GDAL WKT Raster driver (basic version right now, read only. It’s in my TODO list) you can connect WKT Raster with any program that uses GDAL for raster reading, but honestly, I haven’t tested it. It’s planned to work in this point.

EXPORT RASTER DATA FROM ORACLE GEORASTER

Again, we have three ways of exporting data from Oracle GeoRaster:

• Using PL/SQL exporter
• Using Java-based exporter
• Using Java GeoRaster viewer

Export raster data using PL/SQL exporter

As in the importing option, the first step is, as sysdba, grant permissions to our user, and to MDSYS. Write permissions, this time:

call dbms_java.grant_permission(‘JORGE’, ‘SYS:java.io.FilePermission’,
‘C:\orcl_tut\export\fichero.tif’, ‘write’);

call dbms_java.grant_permission(‘MDSYS’, ‘SYS:java.io.FilePermission’,
‘C:\orcl_tut\export\fichero.tif’, ‘write’);

Now, a bit of real PL/SQL again:

DECLARE

geor SDO_GEORASTER;

BEGIN

SELECT image INTO geor FROM spain_images WHERE image_id = 1;

-- Export the whole GeoRaster object into a GeoTIFF file.

sdo_geor.exportTo(geor,NULL, 'GeoTIFF', 'file',

'C:\orcl_tut\export\image_exported_by_plsql');

END;

Problem: Formats accepted for image exporting using this method are only TIFF, BMP, GeoTiff or PNG. If you want JPEG or GIF, you have to use the Java exporter.

And WKT Raster? As for reading, the formats allowed for writting are GDAL-dependent. You can use gdal_translate for exporting data, but as I’ve said, the GDAL WKT Raster driver is still a basic one. Needs more work and testing. Apart from this, there are JPEG and TIFF exporters planned, internally builded, not GDAL dependent.

Executing this code, we get…

GeoRaster exporting error with PL/SQL

… yes, a memory error.

Problem: In the Oracle GeoRaster developers guide, we read that “The maximum amount of GeoRaster data that can be exported in a single operation is 67 megabytes”. Our raster data are greater than this (files of 75MB). But the most strange thing is during the import operation we didn’t have that problem.

And WKT Raster? The export size limits are determined by the file format itself, and the operating system/file system capabilities. This is, again, a GDAL issue. WKT Raster itself doesn’t impose a limitation. Further information here.

Ok, we can export only a fragment of the data. Using the viewer, we get init and end row and column, to see a significative part of our files. Let’s try it again, this time with the cropArea option:

DECLARE

geor SDO_GEORASTER;

BEGIN

SELECT image INTO geor FROM spain_images WHERE image_id = 1;

-- Export the whole GeoRaster object into a GeoTIFF file.

sdo_geor.exportTo(geor, ’cropArea=(2024,951,2831,1663)’, 'GeoTIFF', 'file',

'C:\orcl_tut\export\image_exported_by_plsql’);

END;

The file name without extension. Now the export operation finishes. Here, the result:

GeoRaster image exported with PL/SQL

I personally think  this is a severe limitation. Are 67 MB too much megabytes for Oracle GeoRaster?, why?

Export raster data using Java-based exporter

The Java exporter works in a similar way than the Java importer.

Problem: We have to provide some parameters, like raster data table and raster id, that could be not very intuitive. We have to remember the raster data table name and the raster id we used when loading the data if we want to export the data loaded from the original files.

And WKT Raster? As said, for data exporting, you use gdal_translate. Not well tested yet. But the needed options are clearly explained here

Minor problem: In the README file, by the way, there is a minor typo when explaining the exporter calling method. There’s a repeated parameter in parameter list. Not very important, but it should be fixed in next releases.

Then, the instruction to export one of the raster previously imported is:

java oracle.spatial.georaster.tools.GeoRasterExporter localhost 1521 user password thin
32 T images image "1,rdt1" "GEOTIFF" "export_file.tif"

This time, the file name with extension, not like in the PL/SQL example. The formats accepted are the same than with the Java importer: TIFF, GeoTIFF, JPEG, BMP, GIF, PNG and JP2. After executing this line, we get…

GeoRaster Java exporter memory error

… yes, a memory error again.

Problem: The raster is too big to be exported. The maximum size is, again, 67 MB. And we didn’t have this problem while loading with the Java importer.

But this time, we have another solution, to get the whole raster exported, instead of using the cropArea method: the Java VM -Xmx option.

GeoRaster Java exporter ok with more heap memory

Then, if we increase the maximum heap size for our Java VM, we can get the whole raster data. I think it shouldn’t be necessary. I didn’t find these limitations working with GDAL and PostGIS WKT Raster.

Export raster data using Java GeoRaster viewer

Finally, we can export our data by using the Java GeoRaster Viewer. In the screenshot, the menu options available for doing this operation are marked in red

GeorRaster viewer exporting

Then, we provide some basic parameters (see screenshot)

GeoRaster viewer exporting parameters

Problem: If we choose GeoTIFF as format, we don’t need a World File. The program should inform about this, because while importing with the GeoRaster viewer, we have such option.

Ok, we accept the options and then…

Big problem: Yes, memory problem again. But the thing is the program doesn’t inform us about this problem. Actually, says the data were exported correctly! And the file has not been exported. I think this is an important problem, and must be fixed. See screenshot

GeoRaster viewer exporting error

In case of selecting any different format from GeoTIFF, and tell the viewer to generate a World File, it’s correctly generated, but the image file, as we’ve seen, doesn’t. The parameters of the World File are generated using scientific notation, but are correct.

Of course, we can execute the GeoRaster viewer using -Xmx parameter, or export only an area of the raster, but I think this is not a good solution.

CONCLUSSIONS:

With Oracle GeoRaster, you can have a basic raster-in-database support, but I’m surprised to find bugs, important limitations and non-intuitive tools in an extension of a commercial software like this. Problems that, in most cases, WKT Raster has solved, or never suffered from them.

The Oracle GeoRaster project seems to be something left since 2007. Next days, I’m going to delve into the tool, comparing its functionality with PostGIS WKT Raster, following the Pierre’s tutorial