Skip to content
This repository has been archived by the owner on Jul 24, 2021. It is now read-only.

API trimms sub-second timestamps for some VERY precise tracks from norc.ro #1657

Closed
openstreetmap-trac opened this issue Jul 23, 2021 · 2 comments

Comments

@openstreetmap-trac
Copy link

Reporter: eddyp
[Submitted to the original trac issue database at 10.15am, Wednesday, 18th March 2009]

Hello,

As a result of the discussion in ticket #1648, it appears that the OSM API trimms down the sub-second information for the GPX traces. This some multiple undesired effects (missing information, Potlach breakage).

Here are some images that show how the issue manifests itself in Potlatch:
http://trac.openstreetmap.org/attachment/ticket/1648/gpx-complete-via-edit-trace.png
http://trac.openstreetmap.org/attachment/ticket/1648/gpx-visible-via-edit-trace.png
http://trac.openstreetmap.org/attachment/ticket/1648/interrupted-gpx.png

We, the Romanian team have managed to establish a communication channel with the company behind Norc.ro and they agreed to make available for us the tracks they have used in their featured project norc.ro: See the 28th of February image http://wiki.openstreetmap.org/wiki/Featured_images.

The tracks norc provided are:

  • very precise in positioning with corrections (dgps)
  • very fine grained (5 track points/second)
  • the biggest number of tracks ever imported from a single source (i.e. excluding "Lost GPX Traces") - see the norcTracks user which currently has 15321173 traces: http://www.openstreetmap.org/stats/data_stats.html
  • covering mutiple regions in Romania, Poland, Czech Republic, Slovakia, Austria, Russia

Because of the sub-second trimming, valuable information is missing and Potlatch is not able to "see" the tracks in their entirety. As I understood from the discussions with Richard in http://trac.openstreetmap.org/ticket/1648, the internal table that indexes the gps traces has intentionally trimmed time stamps (removes sub-second time stamps), thus chopping the accurate traces norc.ro provided.

Taking into account the generosity of norc, fact that they are going to provide other traces, too n the future and the fact that is in our best interest to have those numerous accurate tracks correctly present in our databases, could someone analyse this issue and provide a way to fix it?

Is the data type not accurate enough to hold this information? Could the table be altered easily and the data type changed so it allows storing sub-second timestamps? I know that the code around the queries regarding traces might need modification, but is this fesable? As I undestood, the traces are also kept in their original form (which is good); if the data is already broken in the tables, could a massive reimport of the traces from norcTracks be done to fix the issue, once the tables are modified to be able to store that information?

As I understood, the sub-second trimming code was added as a counter-measure against faked traces with identical subsequent time stamps, but, to me, that sounds like the wrong workround for the problem, since it would have been better to simply reject the traces which contained actual time stamp repretitions. How often this thing happened? Do people still try to use/import such traces?

@openstreetmap-trac
Copy link
Author

Author: tom[at]compton.nu
[Added to the original trac issue at 7.30pm, Wednesday, 18th March 2009]

A simple bug report would have done you know - you didn't really need to write a thousand word essay...

First up let me state clearly that this is not any sort of security or privacy thing, so whoever told you that was wrong I'm afraid.

The reason is actually remarkably simple - there is no timestamp type in MySQL with a precision greater than one second. So there really isn't much we can do.

Frankly speaking, sampling five times a second is just ridiculous for our purposes anyway - the only things it's good for is winning a "my trace is bigger than yours competition" of the sort you seem to have engaged in ;-)

@openstreetmap-trac
Copy link
Author

Author: eddyp
[Added to the original trac issue at 8.31pm, Wednesday, 18th March 2009]

Replying to [comment:1 tom[at]compton.nu]:

A simple bug report would have done you know - you didn't really need to write a thousand word essay...

Are you serious? In any other project people are adviced to provide as much info as possible.

Sometimes it is useful to provide a suggestion for the fix. I did both.

First up let me state clearly that this is not any sort of security or privacy thing, so whoever told you that was wrong I'm afraid.

The reason is actually remarkably simple - there is no timestamp type in MySQL with a precision greater than one second. So there really isn't much we can do.

Could the time stamp stored in varchar/string (or whatever is called in MySQL)? That's what I was suggesting with "alter the table".

Frankly speaking, sampling five times a second is just ridiculous for our purposes anyway - the only things it's good for is winning a "my trace is bigger than yours competition" of the sort you seem to have engaged in ;-)

It looks as if you didn't read the report at all. I didn't engage into any pissing contest.

Those are VERY precise tracks from norc, an external company which was kind enough to provide us the biggest source of accurate tracks in our database at this hour (excluding "Lost GPX Traces"). They did those tracks for their streetview project which is accessible at http://norc.ro/ .

Here is an excerpt from http://www.openstreetmap.org/stats/data_stats.html:

Top 50 users for uploads of GPS data
User Number of Points
Lost GPX Traces 54683082
norcTracks 15321173
AkMeR 11980735
RomaniaTracks 8146723
DevineMe 7841325

So, how difficult would it be to alter the table in question to store the time stamps in some sort of string data type?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant