source: subversion/applications/utils/tirex/doc/tirex-overview.pod @ 20995

Last change on this file since 20995 was 20619, checked in by jochen, 10 years ago

tirex doc improvements

File size: 7.8 KB
Line 
1
2=head1 TIREX TILE RENDERING SYSTEM
3
4The Tirex tile rendering system renders map tiles on the fly and caches them.
5It is very similar (and mostly backwards compatible) to the mod_tile/renderd
6system, but improves on it in many respects.
7
8=head2 MAPS
9
10Tirex can handle several maps. Maps can have different styles and potentially
11use different data. In the most common case different maps are just different
12map.xml style files for the Mapnik renderer.
13
14=head2 TILES
15
16As with many other map systems, the map of the world uses a Mercator projection
17and is available in several zoom levels and divided into tiles. Tiles are
18always square and 256x256 pixels. At zoom level 0 the map of the whole world
19fits into one tile, at zoom level 1 there are four (two by two) tiles for the
20whole world. At zoom level 2 there are 16 tiles and so on. The highest zoom
21level depends on the detail you want. For OSM its normally about 18 to 20.
22Tirex does not check the max zoom level.
23
24Tiles are numbered x=0 (-180 degrees) to x=2^zoom-1 (+180 degrees) and from y=0
25(about 85 degrees) to y=2^zoom-1 (about -85 degrees).
26
27=head2 METATILES
28
29The tile size (256x256) is optimized for fast transfer on the web, it is not
30necessarily the best size for handling tiles on the tile server. Rendering many
31small tiles takes more time than rendering fewer bigger tiles and storing many
32small tiles in single files is inefficient because of the block size used by
33filesystems. Also a directory with many files in it will be slower than with
34fewer files.
35
36This is solved by aggregating several tiles into a metatile. Typically 8x8
37tiles make up one metatile, but this is configurable (although sizes other than
388x8 are not well tested) in Tirex. This is the same size as used in
39mod_tile/renderd systems. The tile server internally always "thinks" in
40metatiles. Access requests for tiles are converted into requests for metatiles.
41Metatiles are rendered and stored.
42
43Metatiles are numbered just like tiles, a metatile number is always the same as
44the number of the top-left tile, in other words to get the metatile number you
45have to round the tile coordinates down to the neirest multiple of the metatile
46size (8). Tile (x=17 y=4) is metatile (x=16 y=0).
47
48In the Tirex system metatiles are represented by Tirex::Metatile objects. In
49addition to the x- and y-coordinates and the zoom level, a metatile also needs
50the map name.
51
52=head2 JOBS
53
54If a metatile should be rendered, a job must be created for it. The job
55contains at least the information about the metatile and the priority. It can
56also contain additional information such as the expire time, the source of the
57request and the time it took to render the job (once it is rendered).
58
59In the Tirex system jobs are represented by Tirex::Job objects.
60
61The master will keep track on where a job came from so that it can notify the
62source when the job is done.
63
64=head2 THE QUEUE
65
66The master keeps a prioritized queue of jobs. Every job that comes in will be
67placed on the queue. When there are free renderers, the master will take the
68first job from the queue and sends it to the renderer.
69
70There is only one queue for all jobs, but it is prioritized. Priorities are
71positive integers, 1 is the highest priority. New jobs are added before the
72first job in the queue with a lower priority. Jobs are always taken from the
73front of the queue. So jobs will always be worked on based on their priority
74and age.
75
76A job can have an expire time. If it comes up to be rendered and the expire
77time has passed, the job will not be rendered. This basically allows you to
78say: I have a metatile I want rendered because I might need it in the next few
79minutes. But if its not rendered in, say, 10 minutes, you needn't bother
80rendering it.
81
82When a new job comes in for a metatile that already has a job on the queue,
83those two jobs will be merged. The old job will be taken from the queue and
84a new job will be added. The new job has the higher priority of both jobs. The
85expire time of the new job will be the larger of both times. No expire time is
86the same as "infinite expire time", so if at least one of the jobs has no
87expire time, the new job will have no expire time. Both sources of the two jobs
88will be notified when the job is rendered.
89
90The queue is implemented in a way that its doesn't matter much how many jobs
91there are in the queue. If you want to stick 1000 or 10000 jobs in the
92queue, thats ok.
93
94It is your job as administrator of the system to decide which priorites to use
95for which kind of requests. Live requests coming in from a user should probably
96get a higher priority than batch requests to re-render old tiles. The Tirex
97system gives you the mechanisms needed, you have to decide which jobs get
98priority, how long they should stay on the queue etc.
99
100=head2 PRIORITY CLASSES (BUCKETS)
101
102Tirex allows an infinite number of priorities. To make configuration and
103handling easier, these priorities can be divided up into several priority
104classes (sometimes called buckets). Each priority class has a name and
105represents all priorities in a certain range. You define the name and range.
106Configuration and some other operations will use those priority classes instead
107of the priorities itself.
108
109A typical setup will have a priority class for live requests from the web (lets
110call it 'live') that works on priorities 1 to, say, 9. And then one or more
111priority classes for background requests for higher priorities.
112
113=head2 THE MANAGER
114
115The rendering manager is the heart of the Tirex system. Its job is to work
116throught the queue in order and to dispatch Jobs to be rendered when its
117their turn. The manager takes the configuration and the current system
118load into account when deciding which and how many tiles can be rendered.
119
120=head2 MESSAGES
121
122The Tirex system consists of several processes that work together. Requests for
123the rendering of metatiles and other messages are passed between those
124processes through UDP datagram sockets. Datagram sockets are used to make
125handling of many data sources easier in a non-blocking environment. Because UDP
126sockets can not garantuee delivery of the data over the network but can if you
127used them locally, normal Tirex operation only works on a single host.
128
129Messages always have the same, similar format: Fields are written as
130"key=value" pairs (no spaces allowed), one per line. A linefeed (LF, "\n") is
131used as a line ending, the software ignores an additial carriage return (CR,
132"\r") before the linefeed. Each message must have a "type" (for instance
133"type=metatile_request"). Requests will normally be answered by a reply with
134the same type an added "result" field. The result can either be positive
135("result=ok") or negative ("result=error"). More descriptive error messages are
136also allowed, they always begin with "error_" ("type=error_unknown_message").
137An additional error message for human consumption can be added in the "errmsg"
138field.
139
140=head2 COMPONENTS
141
142The Tirex tile rendering system consists of three major parts:
143
144=over 2
145
146=item * Clients (tirex-batch, mod_tile, ...) want certain tiles rendered. They create rendering requests and send them to the master. If they want to know when a tile is finished rendering, they can wait for an answer, or they can just go on (for batch operations)
147
148=item * The master daemon (tirex-master) runs continuously. It takes requests for tiles from the clients and puts them into a prioritized queue. It works through this queue and sends requests on to the renderer making sure that there are not too many simultaneous renderings going on.
149
150=item * The renderer component (several tirex-renderds) do the actual rendering of the tile, write it to disk and tell the master when they are done.
151
152=back
153
154This gives you an overview of the system, of course the details are a bit more complicated.
155
156=head2 PROGRAMS
157
158Tirex provides the following programs:
159
160=over 8
161
162=item tirex-master
163
164=item tirex-send
165
166=item tirex-status
167
168=item tirex-renderd-starter
169
170=back
171
172
Note: See TracBrowser for help on using the repository browser.