Opened 19 months ago

Last modified 19 months ago

#5468 new defect

Osmosis efficiency problem - suggestion

Reported by: bemysocialdate@… Owned by: bretth
Priority: critical Milestone:
Component: osmosis Version:
Keywords: Cc:

Description

I'm trying to make some more efficient pbf reader, and i had a chance to look at the osmosis one. Although it's not reaching any major release version, but i have spotted two bugs:

1, evaluating DenseNode? has got a programming error: private void processNodes(DenseNodes? nodes, PbfFieldDecoder? fieldDecoder)

Looking at the second cycle, with keysValuesIterator unnecessary loops has been introduced.

The main problem in the file it's massive.

If the keysValuesIterator loop would be removed, and a ComplexEntityData? would be introduced, osmosis would speed up 5 times.

ComplexEntityData? { entities: List<CommonEntityData?> - CommonEntityData? does not hold any tag in this case. tags : List<Tag> }

2, Even if osmosis using some sort of Threading, it's not using all the cores of my processor.

I have refactored the whole thing using RandomAccessFile?, and LinkedBlockingQueue? for task separation.

Using ExecutorService? executorService = Executors.newFixedThreadPool(workers); inside the PbfReader?.run method is a shame.

it should be:

int workers = 4 PbfReader? pbfReader = new PbfReader?(); ExecutorService? executorService = Executors.newFixedThreadPool(); executorService.submit(pbfReader);

I have attached my refactored code:

Some of the classes are superfluous, as previously i didn't know about pbf, and i have recreated some of them, but realized later on.

The example is using all the 4 cores in my CPU, loading a huge map file, which i think on a modern computer is ok.

My purpose was not to recreate the osmosis that's why i haven't finished this work. I had to make some exercise to jump to node.

Attachments (1)

pbfreader.zip (54.1 KB) - added by bemysocialdate@… 19 months ago.
refactored code

Download all attachments as: .zip

Change History (2)

Changed 19 months ago by bemysocialdate@…

Attachment: pbfreader.zip added

refactored code

comment:1 Changed 19 months ago by bemysocialdate@…

First point is irrelevant anymore, as i have misread the code.

I think it's simply an unreadable code. Now i know there is a delimiter between tags of nodes.

Note: See TracTickets for help on using tickets.