Opened 6 years ago

Closed 2 years ago

#4790 closed defect (fixed)

Unable to open europe.osm.pbf

Reported by: errox Owned by: jburgess777@…
Priority: blocker Milestone:
Component: osm2pgsql Version:
Keywords: osm2pgsql, unable to open, big file Cc:

Description

Hello Everybody,

i posted it already in the Forum.

http://forum.openstreetmap.org/viewtopic.php?pid=314287#p314287

No one aswer it.

If i wanted to Import europe.osm.pbf it throws me the error: Unable to open europe.osm.pbf

Germany / belgium alone works fine.

My Command:

osm2pgsql --slim -C 1500 europe.osm.pbf

Change History (4)

comment:1 Changed 6 years ago by jburgess777@…

I suspect you are using a 32 bit VM. It looks like the current code will fail to open a PBF file > 4GB when compiled as a 32 bit executable. It should be possible to fix this but importing a large amount of data on a 32 bit system is probably going to be slow.

I'll try to look at fixing this but in the short term you may be better off trying to run a 64 bit OS instead.

comment:2 Changed 6 years ago by errox

It has to be Possible with an 32 bit OS, too or not?

You are right, it is an 32 bit OS (Ubuntu with 5700 RAM -> VM) and 4 GB Swap.

If i use -1500 isnt he supposed to use only 1500 MB RAM?

comment:3 Changed 6 years ago by amm

It looks like Ubuntu still does not default to using large file support[1] for their 32 bit OS version. So all of the file offset pointers are only 32 bit as well and thus can't handle large files.

It seems just adding a -D_FILE_OFFSET_BITS=64 to the compile flags might solve it. But as each OS and flavour seems to behave differently in this area, I am not sure what the best way to do this in configure is.

That said, even if you can open the file, you might well run into trouble later during the process on a 32 bit OS for such large imports, so I would definitely recommend using a 64 bit OS. Particularly if you say you have more than 4Gb of physical ram!

[1] http://sourceware.org/bugzilla/show_bug.cgi?id=13047

comment:4 Changed 2 years ago by pnorman

Resolution: fixed
Status: newclosed

We've dropped 32 bit support, so this no longer matters, and people are regularly using osm2pgsql with large files.

Note: See TracTickets for help on using tickets.