Opened 10 years ago

Closed 10 years ago

Last modified 10 years ago

#1657 closed defect (wontfix)

API trimms sub-second timestamps for some VERY precise tracks from norc.ro

Reported by: Eddy Petrișor Owned by: Tom Hughes
Priority: major Milestone:
Component: api Version:
Keywords: norcTracks sub-second trimming dgps Cc:

Description

Hello,

As a result of the discussion in ticket #1648, it appears that the OSM API trimms down the sub-second information for the GPX traces. This some multiple undesired effects (missing information, Potlach breakage).

Here are some images that show how the issue manifests itself in Potlatch: http://trac.openstreetmap.org/attachment/ticket/1648/gpx-complete-via-edit-trace.png http://trac.openstreetmap.org/attachment/ticket/1648/gpx-visible-via-edit-trace.png http://trac.openstreetmap.org/attachment/ticket/1648/interrupted-gpx.png

We, the Romanian team have managed to establish a communication channel with the company behind Norc.ro and they agreed to make available for us the tracks they have used in their featured project norc.ro: See the 28th of February image http://wiki.openstreetmap.org/wiki/Featured_images.

The tracks norc provided are:

  • very precise in positioning with corrections (dgps)
  • very fine grained (5 track points/second)
  • the biggest number of tracks ever imported from a single source (i.e. excluding "Lost GPX Traces") - see the norcTracks user which currently has 15321173 traces: http://www.openstreetmap.org/stats/data_stats.html
  • covering mutiple regions in Romania, Poland, Czech Republic, Slovakia, Austria, Russia

Because of the sub-second trimming, valuable information is missing and Potlatch is not able to "see" the tracks in their entirety. As I understood from the discussions with Richard in http://trac.openstreetmap.org/ticket/1648, the internal table that indexes the gps traces has intentionally trimmed time stamps (removes sub-second time stamps), thus chopping the accurate traces norc.ro provided.

Taking into account the generosity of norc, fact that they are going to provide other traces, too n the future and the fact that is in our best interest to have those numerous accurate tracks correctly present in our databases, could someone analyse this issue and provide a way to fix it?

Is the data type not accurate enough to hold this information? Could the table be altered easily and the data type changed so it allows storing sub-second timestamps? I know that the code around the queries regarding traces might need modification, but is this fesable? As I undestood, the traces are also kept in their original form (which is good); if the data is already broken in the tables, could a massive reimport of the traces from norcTracks be done to fix the issue, once the tables are modified to be able to store that information?

As I understood, the sub-second trimming code was added as a counter-measure against faked traces with identical subsequent time stamps, but, to me, that sounds like the wrong workround for the problem, since it would have been better to simply reject the traces which contained actual time stamp repretitions. How often this thing happened? Do people still try to use/import such traces?

Change History (2)

comment:1 Changed 10 years ago by Tom Hughes

Resolution: wontfix
Status: newclosed

A simple bug report would have done you know - you didn't really need to write a thousand word essay...

First up let me state clearly that this is not any sort of security or privacy thing, so whoever told you that was wrong I'm afraid.

The reason is actually remarkably simple - there is no timestamp type in MySQL with a precision greater than one second. So there really isn't much we can do.

Frankly speaking, sampling five times a second is just ridiculous for our purposes anyway - the only things it's good for is winning a "my trace is bigger than yours competition" of the sort you seem to have engaged in ;-)

comment:2 in reply to:  1 Changed 10 years ago by Eddy Petrișor

Replying to tom@compton.nu:

A simple bug report would have done you know - you didn't really need to write a thousand word essay...

Are you serious? In any other project people are adviced to provide as much info as possible.

Sometimes it is useful to provide a suggestion for the fix. I did both.

First up let me state clearly that this is not any sort of security or privacy thing, so whoever told you that was wrong I'm afraid.

The reason is actually remarkably simple - there is no timestamp type in MySQL with a precision greater than one second. So there really isn't much we can do.

Could the time stamp stored in varchar/string (or whatever is called in MySQL)? That's what I was suggesting with "alter the table".

Frankly speaking, sampling five times a second is just ridiculous for our purposes anyway - the only things it's good for is winning a "my trace is bigger than yours competition" of the sort you seem to have engaged in ;-)

It looks as if you didn't read the report at all. I didn't engage into any pissing contest.

Those are VERY precise tracks from norc, an *external* company which was kind enough to provide us the biggest source of accurate tracks in our database at this hour (excluding "Lost GPX Traces"). They did those tracks for their streetview project which is accessible at http://norc.ro/ .

Here is an excerpt from http://www.openstreetmap.org/stats/data_stats.html:

Top 50 users for uploads of GPS data User Number of Points Lost GPX Traces 54683082 norcTracks 15321173 AkMeR 11980735 RomaniaTracks? 8146723 DevineMe? 7841325

So, how difficult would it be to alter the table in question to store the time stamps in some sort of string data type?

Note: See TracTickets for help on using tickets.