Opened 8 years ago

Closed 8 years ago

#3891 closed enhancement (fixed)

implement pgsql_nodes_get_list

Reported by: amm Owned by: jburgess777@…
Priority: minor Milestone:
Component: utils Version:
Keywords: Cc:

Description

Currently pgsql_nodes_get_list is implemented in the naive way of running a for loop through pgsql_nodes_get, resulting in an sql query per node.

This patch implements the function by sending only a single sql query and matching the results back onto the list.

This should be more efficient, although the degree to which it enhances performance will heavily depend on both hardware, the data imported and the cache hit ratio.

Preliminary results indicate it can be a substantial win though in low cache hit regimes.

E.g. importing a berlin extract (17 mb .osm.pbf) on a laptop with a single slow disk reduced the import time from about 5 - 6 mintues down to ~ 3 minutes.

As for normal settings of the cache, one would get 100% hit ratio for such a small extract, it was run with only 8Mb of cache in osm2pgsql (Although with normal disk cache size for postgresql), resulting in cache hit ration of about 20%.

Someone else (Komzpa?) has reported similar numbers on irc of up to 2 times performance improvement from an earlier independent patch that was never submitted.

I have not committed the patch directly to svn, as I currently don't have a full test rig to verify that it handles all the corner cases correctly.

Attachments (1)

patch.diff (4.6 KB) - added by amm 8 years ago.
Patch v1

Download all attachments as: .zip

Change History (4)

Changed 8 years ago by amm

Attachment: patch.diff added

Patch v1

comment:1 Changed 8 years ago by dieterdreist

Component: mapnik-deutils
Owner: changed from mapnik-de@… to Tom Hughes

this patch seems to be for osm2pgsql. Not completely sure which category it should go to.

comment:2 Changed 8 years ago by Tom Hughes

Owner: changed from Tom Hughes to jburgess777@…
Status: newassigned

comment:3 Changed 8 years ago by amm

Resolution: fixed
Status: assignedclosed
Note: See TracTickets for help on using tickets.