Skip to content
This repository has been archived by the owner on Jul 24, 2021. It is now read-only.

Osmosis should sync files to disk during replication #5310

Open
openstreetmap-trac opened this issue Jul 23, 2021 · 2 comments
Open

Osmosis should sync files to disk during replication #5310

openstreetmap-trac opened this issue Jul 23, 2021 · 2 comments

Comments

@openstreetmap-trac
Copy link

Reporter: TomH
[Submitted to the original trac issue database at 8.41pm, Friday, 17th April 2015]

If a machine that is running replication crashes then, with a modern logging file system, it is likely that when it reboots the most recent file will exist but without a contents.

That can be fixed by deleting those files so that replication restarts from the previous state, but any client which had already downloaded the old version before the crash will miss some edits, as the new version of that file will be larger and won't be downloaded again.

To minimise this risk it would be good to sync the files after they are created.

@openstreetmap-trac
Copy link
Author

Author: mmd
[Added to the original trac issue at 5.57pm, Tuesday, 5th May 2015]

Adding a few comments from a data consumer point of view:

According to the [http://wiki.openstreetmap.org/wiki/Overpass_API/status Overpass API status page], the data replication was hit by this bug at least twice this year. Generating the same diff file a second time with different contents will usually go unnoticed, causing later database inconsistency issues and generally loss of data (changesets only contained in the second diff write are lost).

As of today, this can only be resolved by time consuming manual repair activities or even restoring a backup and reapplying minutely diffs, during which Overpass API will serve outdated information.

It would be really helpful to have this issue analyzed in more detail and possibly even fixed. Dealing with empty diff files is not a concern. In the current case, the diff file was initially written with a size of 48375 bytes and again after the crash with a size of 2.2M.

Please also refer to this Github ticket for more details: drolbr/Overpass-API#210

@openstreetmap-trac
Copy link
Author

Author: Jocelyn
[Added to the original trac issue at 6.25pm, Sunday, 3rd July 2016]

Wouldn't a better fix be to write temporary files before moving them to the final location ? Move operation is supposed to be an atomic operation, and if the .osm.gz is moved before the state.txt file, then no issue should be seen anymore.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant