Opened 3 years ago

Closed 19 months ago

#5406 closed defect (fixed)

osmosis 0.45 read-pgsql error: java.lang.ClassCastException: org.postgresql.util.PGobject cannot be cast to java.util.Map:

Reported by: Piskvor Owned by: bretth
Priority: major Milestone:
Component: osmosis Version:
Keywords: Cc:

Description

On a Ubuntu 12.04 box, with Oracle Java 8 (1.8.0_91-b14, 64-bit), I have a local copy of Czech Republic OSM data, updated from http://download.geofabrik.de/europe/czech-republic-updates via --rri and stored in pgSQL 9.1 database; have been successfully building maps from that (i.e. this part works).

Config:

baseUrl=http://download.geofabrik.de/europe/czech-republic-updates
maxInterval=1800

Sequence state as of now:

#Tue Jun 14 08:20:38 CEST 2016
sequenceNumber=1186
timestamp=2016-06-12T20\:29\:02Z

Updating through this command:

osmosis --rri workingDirectory=/home/foobar/.osmosis-minutely/ --sc --wpc user="osmosis" database="osmosis" password="hunter2"

A few days ago, I have updated osmosis from 0.44.1 to 0.45; built a few maps in the meantime, all worked. Today, after the rri update, I started getting a ClassCastException? when reading the database, after about 2 real-minutes of runtime (for a task which usually finishes under a minute). When using 0.44.1-SNAPSHOT (built from commit 90770d38e452678ac672a43d0182432b46925e0a), I never get the exception; when using 0.45, I always get it. The maps generated using 0.44.1 seem to be working correctly.

osmosis --read-pgsql database=osmosis user=osmosis --dataset-bounding-box bottom=49.79 top=50.33 left=14.07 right=14.86 completeWays=no --buffer --write-xml file=./praha-extract.osm
Jun 15, 2016 9:31:56 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.45
Jun 15, 2016 9:31:56 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
Jun 15, 2016 9:31:56 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Launching pipeline execution.
Jun 15, 2016 9:31:56 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline executing, waiting for completion.
[Jun 15, 2016 9:34:31 AM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 1-read-pgsql failed
java.lang.ClassCastException: org.postgresql.util.PGobject cannot be cast to java.util.Map
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.EntityRowMapper.mapCommonEntityData(EntityRowMapper.java:81)
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.NodeRowMapper.mapRow(NodeRowMapper.java:30)
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.NodeRowMapper.mapRow(NodeRowMapper.java:17)
        at org.openstreetmap.osmosis.pgsnapshot.common.RowMapperRowCallbackListener.processRow(RowMapperRowCallbackListener.java:46)
        at org.springframework.jdbc.core.JdbcTemplate$RowCallbackHandlerResultSetExtractor.extractData(JdbcTemplate.java:1566)
        at org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:459)
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:404)
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:470)
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:475)
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.EntityDao.getFeaturelessEntity(EntityDao.java:178)
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.EntityDao.iterate(EntityDao.java:227)
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.PostgreSqlDatasetContext.iterateBoundingBox(PostgreSqlDatasetContext.java:406)
        at org.openstreetmap.osmosis.dataset.v0_6.DatasetBoundingBoxFilter.process(DatasetBoundingBoxFilter.java:80)
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.PostgreSqlDatasetReader.run(PostgreSqlDatasetReader.java:53)
        at java.lang.Thread.run(Thread.java:745)

Jun 15, 2016 9:34:31 AM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion

[...errors cascading down the pipeline omitted...]

real    2m35.758s
user    0m1.128s
sys     0m0.116s

Tried a different bounding box, same results: works with 0.44.1, not with 0.45:

osmosis --read-pgsql database=osmosis user=osmosis --dataset-bounding-box bottom=49.00 top=49.30 left=16.40 right=16.90 completeWays=yes --write-xml brno-0.45.osm
Jun 15, 2016 11:08:43 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.45
Jun 15, 2016 11:08:43 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
Jun 15, 2016 11:08:43 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Launching pipeline execution.
Jun 15, 2016 11:08:43 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline executing, waiting for completion.
Jun 15, 2016 11:09:05 AM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 1-read-pgsql failed
java.lang.ClassCastException: org.postgresql.util.PGobject cannot be cast to java.util.Map
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.EntityRowMapper.mapCommonEntityData(EntityRowMapper.java:81)
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.NodeRowMapper.mapRow(NodeRowMapper.java:30)
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.NodeRowMapper.mapRow(NodeRowMapper.java:17)
        at org.openstreetmap.osmosis.pgsnapshot.common.RowMapperRowCallbackListener.processRow(RowMapperRowCallbackListener.java:46)
        at org.springframework.jdbc.core.JdbcTemplate$RowCallbackHandlerResultSetExtractor.extractData(JdbcTemplate.java:1566)
        at org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:459)
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:404)
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:470)
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:475)
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.EntityDao.getFeaturelessEntity(EntityDao.java:178)
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.EntityDao.iterate(EntityDao.java:227)
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.PostgreSqlDatasetContext.iterateBoundingBox(PostgreSqlDatasetContext.java:406)
        at org.openstreetmap.osmosis.dataset.v0_6.DatasetBoundingBoxFilter.process(DatasetBoundingBoxFilter.java:80)
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.PostgreSqlDatasetReader.run(PostgreSqlDatasetReader.java:53)
        at java.lang.Thread.run(Thread.java:745)

The equivalent process using a PBF extract of the same data finishes successfully, and the results seem to be functionally identical:

osmosis-0.45 --read-pbf-fast workers=8 file=./czech-republic-latest.osm.pbf --log-progress --bounding-box bottom=49.00 top=49.30 left=16.40 right=16.90 completeWays=yes --write-xml brno-0.45.osm

Files generated via 0.44.1 can be downloaded at http://maps.piskvor.org/ - nothing unusual in the daily updates or in the generated data, as far as I can tell.

Change History (10)

comment:1 Changed 3 years ago by bretth

Thanks for the report. Surprising error, I'm not sure what's changed in this space. I'll look into it as soon as I can find time.

comment:2 Changed 3 years ago by bretth

I can't reproduce this error. Perhaps there is something unusual in the database.

In order to diagnose the problem, can you please make the following code modification, recompile, and try again? It will check the type of the tags field and throw an exception containing the data type received and the entity id being processed. Hopefully it will help us narrow down the problem.

Insert the following code at line 81 of file osmosis-pgsnapshot/src/main/java/org/openstreetmap/osmosis/pgsnapshot/v0_6/impl/EntityRowMapper.java

Object rawDbTags = rs.getObject("tags");
if (!(rawDbTags instanceof Map)) {
	throw new OsmosisRuntimeException("Invalid tags field. EntityId=" + rs.getLong("id")
		+ " TagsType=" + rawDbTags.getClass().getName());
}
Last edited 3 years ago by bretth (previous) (diff)

comment:3 Changed 3 years ago by bretth

From #5407, perhaps this is postgres version related. That defect also occurs on postgres 9.1. I can't reproduce it on postgres 9.3.9.

One difference between 0.44.1 and 0.45 is the postgres JDBC driver version. Are you able to try recompiling with the older postgres JDBC driver?

Modify gradle.properties and modify the following line:

dependencyVersionPostgreSql=9.4-1204-jdbc4

to be

dependencyVersionPostgreSql=9.4-1201-jdbc4

Then recompile (i.e. ./gradlew assemble).

comment:4 in reply to:  3 ; Changed 3 years ago by jjoe64

I have the same issue.

When reading from pgsql I get that error with current version (0.45 or current git branch). I also tried to change the version of postgres driver like mentioned by brett@ but no effect.

This is my command:

package/bin/osmosis --read-pgsql database=osm user=openstreetmap password='openstreetmap' outPipe.0=pg --dataset-bounding-box inPipe.0=pg left=49.0239 bottom=8.3709 right=49.0389 top=8.4108 outPipe.0=dd --write-xml inPipe.0=dd file=../jaaa.osm

When I checkout the v0.44.1 I works perfectly.

comment:5 in reply to:  4 Changed 3 years ago by bretth

This comment by Jocelyn on ticket 5407 might be the key to your problem: https://trac.openstreetmap.org/ticket/5407#comment:7

It seems to be related to the postgis version. Osmosis is using a postgis 2 library. Are you still using postgis 1.x?

comment:6 Changed 2 years ago by sjones925

I have this issue as well with Osmosis 0.45 and --read-pgsql. When I switch to Osmosis 0.44.1, things work as expected, but I lose the postgresSchema option. Here are my versions:

PostgreSQL 9.5.5 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2, 64-bit
POSTGIS="2.3.0 r15146" GEOS="3.4.2-CAPI-1.8.2 r3921" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.10.1, released 2013/08/26" LIBXML="2.9.1" LIBJSON="0.11.99" RASTER

And the stacktrace:

SEVERE: Thread for task 1-read-pgsql failed
java.lang.ClassCastException: org.postgresql.util.PGobject cannot be cast to java.util.Map
	at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.EntityRowMapper.mapCommonEntityData(EntityRowMapper.java:81)
	at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.WayRowMapper.mapRow(WayRowMapper.java:30)
	at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.WayRowMapper.mapRow(WayRowMapper.java:18)
	at org.openstreetmap.osmosis.pgsnapshot.common.RowMapperRowCallbackListener.processRow(RowMapperRowCallbackListener.java:46)
	at org.springframework.jdbc.core.JdbcTemplate$RowCallbackHandlerResultSetExtractor.extractData(JdbcTemplate.java:1566)
	at org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:459)
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:404)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:470)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:475)
	at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.EntityDao.getFeaturelessEntity(EntityDao.java:178)
	at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.EntityDao.iterate(EntityDao.java:227)
	at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.EntityDao.iterate(EntityDao.java:256)
	at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.PostgreSqlDatasetContext.iterate(PostgreSqlDatasetContext.java:210)
	at org.openstreetmap.osmosis.dataset.v0_6.DumpDataset.process(DumpDataset.java:48)
	at org.openstreetmap.osmosis.pgsnapshot.v0_6.PostgreSqlDatasetReader.run(PostgreSqlDatasetReader.java:53)
	at java.lang.Thread.run(Thread.java:745)

Mar 08, 2017 8:41:45 PM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed.
	at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
	at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
	at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
	at org.codehaus.classworlds.Launcher.main(Launcher.java:47)
Last edited 2 years ago by sjones925 (previous) (diff)

comment:7 Changed 2 years ago by sjones925

If anyone else runs into this, my colleague found that removing lib/default/postgresql-9.1-901-1.jdbc4.jar jar solved this problem. There's another, newer version in the same folder, and keeping that one made the issue go away.

comment:8 Changed 19 months ago by aviv.rosenberg@…

Ran into the same issue with 0.45. The above comments helped me. After downgrading osmosis to 0.44.1 the problem was resolved.

It seems that in version 0.44.1, an old postgresql client library came back, causing this problem:

[aviv@stark:~/Downloads/osm]
% ll /usr/local/Cellar/osmosis/0.43/libexec/lib/default/postgres*
-rw-rw-r--@ 1 aviv  wheel   538K Apr  2  2013 /usr/local/Cellar/osmosis/0.43/libexec/lib/default/postgresql-9.1-901-1.jdbc4.jar
[aviv@stark:~/Downloads/osm]
% ll /usr/local/Cellar/osmosis/0.44.1/libexec/lib/default/postgres*
-rw-rw-r--  1 aviv  admin   633K Jun  4  2015 /usr/local/Cellar/osmosis/0.44.1/libexec/lib/default/postgresql-9.4-1201-jdbc4.jar
[aviv@stark:~/Downloads/osm]
% ll /usr/local/Cellar/osmosis/0.45/libexec/lib/default/postgres*
-rw-rw-r--  1 aviv  admin   538K May 27  2016 /usr/local/Cellar/osmosis/0.45/libexec/lib/default/postgresql-9.1-901-1.jdbc4.jar
-rw-rw-r--  1 aviv  admin   641K May 27  2016 /usr/local/Cellar/osmosis/0.45/libexec/lib/default/postgresql-9.4-1204-jdbc4.jar

comment:9 Changed 19 months ago by bretth

This issue should be resolved in 0.46 ... finally. The duplicate postgresql libraries no longer exist.

comment:10 Changed 19 months ago by bretth

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.