Opened 10 years ago

Closed 10 years ago

Last modified 10 years ago

#1648 closed enhancement (fixed)

Potlatch doesn't show some gpx-es entirely or shows them partially

Reported by: Eddy Petrișor Owned by: richard@…
Priority: minor Milestone:
Component: potlatch (flash editor) Version:
Keywords: gpx missing norc Cc:

Description

Due to our colaboration with Norc (norc.ro) we, the Romanian team, have managed to get the tracks used they got during their streetview surveys. All of these were imported under the user norcTracks:

http://www.openstreetmap.org/user/norcTracks/traces

There seems to be a problem with these tracks and some parts of them are missing entirely or sections of them.

For instance, when trying to edit this area:

http://www.openstreetmap.org/edit?lat=44.83865&lon=24.897446&zoom=18

on the right side, you can see part of a track (322826) from norc, but not as a continuous line.

This is not as bad as the long(/entire?) missing part of the gpx 322818 which should be visible in the centre of the image, between "Strada Basarabiei" (now way 23683847) and "Petrochimistilor" (now way 23330467).

The entire track still can be seen if editing via the edit link in the trace page (I hacked the link to focus by default on the problematic areas):

http://www.openstreetmap.org/edit?gpx=322818&lat=44.8389&lon=24.8979&zoom=18

Same is true for the interrupted track (same hack applied) which was interrupted in normal edit, after 'g', but now is a continuous line:

http://www.openstreetmap.org/edit?gpx=322826&lat=44.83865&lon=24.897446&zoom=18

Why are parts of these tracks missing?

Attachments (3)

gpx-complete-via-edit-trace.png (203.4 KB) - added by Eddy Petrișor 10 years ago.
trace complete via edit
gpx-visible-via-edit-trace.png (232.3 KB) - added by Eddy Petrișor 10 years ago.
trace 322818 is visible via trace edit
interrupted-gpx.png (204.9 KB) - added by Eddy Petrișor 10 years ago.
322818 is not visible at all, 322826 is interrupted via normal edit

Download all attachments as: .zip

Change History (10)

comment:1 Changed 10 years ago by Richard

Priority: majorminor
Type: defectenhancement

The OSM database doesn't import sub-second timestamps on GPS tracks: anything after the second will be truncated.

Potlatch's GPS display (when pressing 'g' - swf_controller.rb) draws lines between points by timestamp. However, where two points have the same timestamp - as these do after the sub-second portion has been truncated - it won't draw a line. This was added to avoid the case where people upload GPS traces with the same (faked) timestamp on every point, which otherwise makes an almighty mess of the display.

I guess the best solution would be for Potlatch to only draw a dot, not a line, in this case.

Changed 10 years ago by Eddy Petrișor

trace complete via edit

Changed 10 years ago by Eddy Petrișor

trace 322818 is visible via trace edit

Changed 10 years ago by Eddy Petrișor

Attachment: interrupted-gpx.png added

322818 is not visible at all, 322826 is interrupted via normal edit

comment:2 in reply to:  1 ; Changed 10 years ago by Eddy Petrișor

Replying to Richard:

The OSM database doesn't import sub-second timestamps on GPS tracks: anything after the second will be truncated.

That is really a bad decision. The tracks which we have from Norc are sampled at 5 samples/second and they have corrections (dgps) so they are really accurate.

Is there a way to fix this now, or do the tables need to altered? The downloaded trace contains the subseconds, so the data is correctly stored, so I guess all the interfaces need to be fixed (since some are working - see edit trace).

Potlatch's GPS display (when pressing 'g' - swf_controller.rb) draws lines between points by timestamp. However, where two points have the same timestamp - as these do after the sub-second portion has been truncated - it won't draw a line. This was added to avoid the case where people

OK, but why are entire long sections missing (see trace 322818 in the snapshot)?

upload GPS traces with the same (faked) timestamp on every point, which otherwise makes an almighty mess of the display.

I think that someone used an unappropriately sized hammer to fix a issue which doesn't seem hard to fix otherwise (find *identically* timed points and eliminate/ignore them).

This sounds fixable, can it be done?

I guess the best solution would be for Potlatch to only draw a dot, not a line, in this case.

That would not be a good idea for traces which intersect themselves since the direction of a connection is not clear at all if only dots are drawn.

I think the best solution is to actually allow such high resolutions, not allow points with the same timestamp and use the same trace rendering path for regular edit as is used when 'trace edit'-ing.

Note that I disagree this is a minor bug or even an enhancement since clearly valid and acurate real data is truncated. I will change this, unless some reasonable explanation for the priority downgrade *and* type downgrade is provided since it helps nobody to hide the dirt under the rug.

comment:3 in reply to:  2 ; Changed 10 years ago by Richard

Replying to eddyp:

Replying to Richard:

The OSM database doesn't import sub-second timestamps on GPS tracks: anything after the second will be truncated.

That is really a bad decision. The tracks which we have from Norc are sampled at 5 samples/second and they have corrections (dgps) so they are really accurate.

Is there a way to fix this now, or do the tables need to altered? The downloaded trace contains the subseconds, so the data is correctly stored, so I guess all the interfaces need to be fixed (since some are working - see edit trace).

No, the data is not correctly stored.

Traces are recorded in two ways on the server. Firstly, they're imported into a table of GPX points. To the best of my knowledge, this truncates timestamps to the second. Potlatch uses this table when you press 'g', because the table is indexed so you can query it by bbox.

Secondly, the original .gpx file is also saved on the server, unchanged (except potentially for .gz compression to save space). Potlatch uses these files when you click 'edit' by a trace. Consequently it is able to retrieve the subsecond timestamp information.

OK, but why are entire long sections missing (see trace 322818 in the snapshot)?

Don't know off-hand. If you can extract the right bit of the trace I'll have a look at it, but I'm on a mobile connection right now and can't download a 5Mb GPX.

I think that someone used an unappropriately sized hammer to fix a issue which doesn't seem hard to fix otherwise (find *identically* timed points and eliminate/ignore them).

This sounds fixable, can it be done?

No, for the above reason.

I think the best solution is to actually allow such high resolutions, not allow points with the same timestamp

Maybe so, but you're talking to the wrong guy here. Potlatch is doing the best it can with the data that exists in the database. I don't know anything about the GPX importer, so I suggest you re-enter/refile as a ticket against the API.

comment:4 in reply to:  3 Changed 10 years ago by Eddy Petrișor

Replying to Richard:

Replying to eddyp:

Replying to Richard:

The OSM database doesn't import sub-second timestamps on GPS tracks: anything after the second will be truncated.

That is really a bad decision. The tracks which we have from Norc are sampled at 5 samples/second and they have corrections (dgps) so they are really accurate.

Is there a way to fix this now, or do the tables need to altered? The downloaded trace contains the subseconds, so the data is correctly stored, so I guess all the interfaces need to be fixed (since some are working - see edit trace).

No, the data is not correctly stored.

Traces are recorded in two ways on the server. Firstly, they're imported into a table of GPX points. To the best of my knowledge, this truncates timestamps to the second. Potlatch uses this table when you press 'g', because the table is indexed so you can query it by bbox.

Secondly, the original .gpx file is also saved on the server, unchanged (except potentially for .gz compression to save space). Potlatch uses these files when you click 'edit' by a trace. Consequently it is able to retrieve the subsecond timestamp information.

Is it possible to get the GPX ids via the indexed table then fetch the complete gpx-es? Would that be too slow?

OK, but why are entire long sections missing (see trace 322818 in the snapshot)?

Don't know off-hand. If you can extract the right bit of the trace I'll have a look at it, but I'm on a mobile connection right now and can't download a 5Mb GPX.

I just tried, Firefox went into an IO spiral.

I think that someone used an unappropriately sized hammer to fix a issue which doesn't seem hard to fix otherwise (find *identically* timed points and eliminate/ignore them).

This sounds fixable, can it be done?

No, for the above reason.

Is the table limited that way? I mean are the data types for the time stamps not fit to store the information, if that limitation were to e removed?

I think the best solution is to actually allow such high resolutions, not allow points with the same timestamp

Maybe so, but you're talking to the wrong guy here. Potlatch is doing the best it can with the data that exists in the database. I don't know anything about the GPX importer, so I suggest you re-enter/refile as a ticket against the API.

Yes, indeed, taking into account that norcTracks is the account with the second most number of tracks (after LostTracks?) which are VERY precise I guess it would make sense to try to fix that in the API, or at least allow a reimport (from the .gz-ed files) without the no-subsecond filter.

comment:5 Changed 10 years ago by Eddy Petrișor

Submitted bug against the API:

http://trac.openstreetmap.org/ticket/1657

comment:6 Changed 10 years ago by Richard

Resolution: fixed
Status: newclosed

Now that we have Postgres, I believe Potlatch itself will order correctly by partial timestamps, so we can close this ticket pending any change or otherwise in the API.

comment:7 Changed 10 years ago by Tom Hughes

Note that only tracks imported since the move to API 0.6 will have fractional timestamps in the points table.

Note: See TracTickets for help on using tickets.