Skip to content
This repository has been archived by the owner on Jul 24, 2021. It is now read-only.

Osmosis efficiency problem - suggestion #5466

Open
openstreetmap-trac opened this issue Jul 23, 2021 · 1 comment
Open

Osmosis efficiency problem - suggestion #5466

openstreetmap-trac opened this issue Jul 23, 2021 · 1 comment

Comments

@openstreetmap-trac
Copy link

Reporter: bemysocialdate[at]gmail.com
[Submitted to the original trac issue database at 8.04am, Wednesday, 30th August 2017]

I'm trying to make some more efficient pbf reader, and i had a chance to look at the osmosis one.
Although it's not reaching any major release version, but i have spotted two bugs:

1, evaluating DenseNode has got a programming error:
private void processNodes(DenseNodes nodes, PbfFieldDecoder fieldDecoder)

Looking at the second cycle, with keysValuesIterator unnecessary loops has been introduced.

The main problem in the file it's massive.

If the keysValuesIterator loop would be removed, and a ComplexEntityData would be introduced, osmosis would speed up 5 times.

ComplexEntityData
{
entities: List - CommonEntityData does not hold any tag in this case.
tags : List
}

2, Even if osmosis using some sort of Threading, it's not using all the cores of my processor.

I have refactored the whole thing using RandomAccessFile, and LinkedBlockingQueue for task separation.

Using
ExecutorService executorService = Executors.newFixedThreadPool(workers); inside the PbfReader.run method is a shame.

it should be:

int workers = 4
PbfReader pbfReader = new PbfReader();
ExecutorService executorService = Executors.newFixedThreadPool();
executorService.submit(pbfReader);

I have attached my refactored code:

Some of the classes are superfluous, as previously i didn't know about pbf, and i have recreated some of them, but realized later on.

The example is using all the 4 cores in my CPU, loading a huge map file, which i think on a modern computer is ok.

My purpose was not to recreate the osmosis that's why i haven't finished this work. I had to make some exercise to jump to node.

@openstreetmap-trac
Copy link
Author

Author: bemysocialdate[at]gmail.com
[Added to the original trac issue at 6.43pm, Saturday, 2nd September 2017]

First point is irrelevant anymore, as i have misread the code.

I think it's simply an unreadable code. Now i know there is a delimiter between tags of nodes.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant