rawr + Metatile generation assistance

[guyisra]

I'm having trouble understanding the flow of commands needed to produce metatiles.
tileserver runs successfully, serving tiles from either tile store (local folder via tilequeue process) or directly from postgres.

From what I understand (and please correct me where I'm wrong):

• once data is in postgres, there are 2 flows:

  1. zoom <= 10
  2. zoom > 10

• for zoom <= 10, the tiles generation would be either with batch or on-demand to generate, store (in s3 or similar) and serve as is, via cdn and use caching (cdn or redis) where appropriate

• for zoom > 10, rawr files needs to be created from which the tiles will be generated

rawr tiles, from what I gather, are created via its own queue (differnet from the tilequeue process queue) and are seeded from the TOI?

• what command(s) are needed to generate them? tilequeue rawr-process produces a admin_areas relation does not exist and tilequeue-rawr-seed-toi explodes after some tiles due to the assert len(coords) == 1

I noticed a (undocumented) command for metatile and meta-tile-low-zoom which seems to fit the process described above but these command are a per tile commands, and do not read from a queue (which I'm not sure how the queue is populated, and by what logic and by who)

thanks

[zerebubuth]

Apologies for the confusion! We've changed how this works a few times, and we're still settling into a stable and documented way to do tile generation. A lot of this is still work in progress while we iterate towards something more automated.

You understand correctly about the split between low (zoom < 10) and high (zoom >= 10) tiles. Our current process (as used for the Nextzen tiles) is to batch generate all the tiles at all zooms. We've started trying to automate that with a package we call tileops. Although it's very much tied to AWS, it might be possible to adapt to other platforms.

An outline of the process is:

1. Import data into the database.
2. Generate RAWR tiles using tilequeue rawr-tile.
3. Generate high-zoom meta tiles from the RAWR tiles using tilequeue meta-tile.
4. Generate low-zoom meta tiles from the database using tilequeue meta-tile-low-zoom.

For the database, we use AWS RDS, and "replicate" by taking a snapshot and starting several servers from the snapshot. For the tile generation (RAWR, high and low meta), we use AWS Batch. Tiles are stored in AWS S3 buckets.

Although all the logic in tileops is to deal with AWS, the general pattern should be applicable to anything that can run databases, batch tasks and has a file store.

Hope that helps!

[guyisra]

Thanks!
so in AWS batch, you define the 3 jobs (rawr, meta-tile, low-zoom) and then what runs the scripts to set everything in motion? since the rawr-tile and other jobs require a --tile option to be passed in. It is currently a Ref::tile which is passed to the job. But by who?

do you have a ecs app that runs the tileops scripts to orchestra the jobs?

[zerebubuth]

At the moment, we've just been running the scripts from our laptops. It's not ideal, and hopefully at some point soon we'll bundle the "control" parts into a container which can be kicked off and monitored from a web page.

The Ref::tile is substituted by AWS Batch based on the value we enqueued the job with (the run_id is just a string which gets echoed to the logs to help us distinguish between multiple runs). It's worth noting that we've been using a queue_zoom of 7 in order to keep the number of jobs down (4**7 = 16k rather than 4**10 = 1M).

[guyisra]

So as I understand, using the rawr-tile job, means to enqueue the job (manually? with a loop?) with all of the tiles that would be in the sqs?

would running rawr-tile --tile z/x/y be equivalent to running rawr-process from the queue after rawr-seed?

[zerebubuth]

The batch-enqueue command handles enqueuing the Batch jobs for rawr-tile. By default (if --file, --tile and --pyramid are not specified) it enqueues all the tiles.

Making RAWR tiles through Batch + rawr-tile or SQS + rawr-process should be equivalent, yes. Although we've only been using Batch recently.

[nvkelso]

Please let us know if you have any other questions with a new issue :)