Missing Tiles

[meteoDaniel]

Dear all I am observing a strange behaviour. Serving tiles dynamically will lead to some missing tiles. After a second attempt to request a tile it is available.

For me it sounds like a problem during concurrency jobs.

Do you have any notice about such a behaviour?

Hopefully there is a solution.

[vincentsarago]

@meteoDaniel do you have any logs or data we can see ?

also what kind of infra you are running this ?

[meteoDaniel]

Happens on AWS as well as I run this locally or an dedicated server. I will prepare an example.

[meteoDaniel]

Here is my code:

import asyncio
import math
import os
from concurrent.futures import ProcessPoolExecutor
from datetime import datetime
from pathlib import Path as posix_path
from typing import Callable, Dict, Type, Tuple, Union

import numpy as np
import fire
from fastapi import Path
from morecantile import TileMatrixSet
from rio_tiler.io import BaseReader, COGReader
from titiler.core.dependencies import (
    ImageParams,
    MetadataParams,
    TMSParams,
    DefaultDependency
)
from titiler.core.resources.enums import ImageType

from tile_server.routes import store_file

class LocalTilerFactory:
    """ Factory to handle incoming tiling requests """

    # Default reader is set to COGReader
    reader: Type[BaseReader] = COGReader

    # Endpoint Dependencies
    metadata_dependency: Type[DefaultDependency] = MetadataParams
    img_dependency: Type[DefaultDependency] = ImageParams

    # TileMatrixSet dependency
    tms_dependency: Callable[..., TileMatrixSet] = TMSParams

    def tile(
            self,
            z: int = Path(..., ge=0, le=30, description="Tiles's zoom level"),
            x: int = Path(..., description="Tiles's column"),
            y: int = Path(..., description="Tiles's row"),
            dt: datetime = datetime(2020, 1, 1),
    ):

        """Create map tile from a dataset."""
        local_file = posix_path(f"{LOCAL_FILE_PATH}/{dt.strftime('%Y%m%d%H%M')}.tif")
        if ~local_file.is_file():
            asyncio.new_event_loop().run_until_complete(store_file(
                f"{BASE_AWS_PATH}{dt.strftime('%Y%m%d%H%M')}.tif",
                local_file))
            src_path = str(local_file)
        else:
            src_path = str(local_file)

        # try:
        with self.reader(src_path) as src_dst:
            data = src_dst.tile(
                x,
                y,
                z,
            )
            dst_colormap = getattr(src_dst, "colormap", None)
        # except:
        #     raise HTTPException(status_code=404, detail="Image not found")
        format = ImageType.jpeg if data.mask.all() else ImageType.png

        image = data.post_process(
        )

        content = image.render(
            img_format=format.driver,
            colormap=dst_colormap,
            **format.profile,
        )

        return content


def process_local_tiling_store(dt: Union[str, datetime]) -> None:
    """ handles tiling and storing of tilePyramid images """
    if isinstance(dt, str):
        dt = datetime.strptime(dt, DATE_TIME_PARSING_FORMAT)

    required_tiles = required_tiles_per_zoom(
        np.array([DEFAULT_BOUNDS[1], DEFAULT_BOUNDS[3]]),
        np.array([DEFAULT_BOUNDS[0], DEFAULT_BOUNDS[2]]),
        5,
        9,
    )
    list_args = []

    os.makedirs(f"{LOCAL_FILE_PATH}/{dt.strftime('%Y%m%d%H%M')}/", exist_ok=True)
    for z in list(required_tiles.keys()):
        for x in required_tiles[z]['x']:
            for y in required_tiles[z]['y']:
                list_args.append((dt, z, x, y))

    with ProcessPoolExecutor(max_workers=4) as executor:
        executor.map(run_tiling_and_store, list_args)


def run_tiling_and_store(argument_tuple: Tuple[datetime, int, int, int]) -> None:
    """ wraps LocalTilerFactory to tile COG image into tile and store locally """
    content = LocalTilerFactory().tile(dt=argument_tuple[0], x=argument_tuple[2],
                                       y=argument_tuple[3], z=argument_tuple[1])
    with open(f"{LOCAL_FILE_PATH}/{argument_tuple[0].strftime('%Y%m%d%H%M')}/"
              f"{argument_tuple[1]}_{argument_tuple[2]}_{argument_tuple[3]}.png", 'wb') as file:
        file.write(content)

I use the TilerFactory to store the TileImages locally to improve access speed. Dynmaic Tiling (inclduing caching) is always slower than using the following API:

@app.get("/radar", tags=["radar"])
async def get_radar_tiles(
        dt: str = Query(..., title="Timestamp", description="Timestamp has to be passed in ISO 8601 "
                                                            "format in UTC. E.g.: "
                                                            "2020-10-06T15:15:00+00:00 %Y-%m-%dT%H:%H:%S%z"),
        x: int = Query(...),
        y: int = Query(...),
        z: int = Query(...),

):
    """API callback to return a single """
    timestamp = datetime.strptime(dt, DATE_TIME_PARSING_FORMAT)
    return FileResponse(f"{LOCAL_FILE_PATH}/{timestamp.strftime('%Y%m%d%H%M')}/{z}_{x}_{y}.png")

It serves the tiles that I have created and sotred locally.

Actually I have to run the creating of the tiles multiple times be sure that all tiles are created.

[vincentsarago]

I don't fully understand what you are trying to do there 🤷‍♂️

are you using titiler at all ?

[meteoDaniel]

@vincentsarago I started to run a tileserver with caching https://developmentseed.org/titiler/examples/code/tiler_with_cache/
and in parallel I create COG files of weather radar data.

Then is use the tile server and the COG files in a leaflet client with TimeDimension extension to create a radar movie. The result was very bad. The titiler server was not able to provide the required tiles fast enough and at least some of the tiles are missing. I have configured a delay of 1-2s between the images.

So I started to find another way to serve the tiles. This is why I moved back to a tile pyramid that is stored on the file system of my tile server.
Actually I create the COG file and afterwards I use titiler to create the tile pyramides. Then I use FastAPI to grant access to the tiles (this is my tile server at the moment).

If I loop through all required tiles (see process_local_tiling_store) . The tilerFactory fails on approx. 20% of the requested tiles. So the solution in the moment is to re run this process 4 times. Then all tiles were created. I had the same behaviour in my first solution with dynamic tiling.

And now I want to know how this could happen? I am not receiving any error message.

[vincentsarago]

Well without data nor logs ... I have absolutely no idea what's going on I'm sorry.

[meteoDaniel]

I have added the dependencies to my code above.
And I have attached an example file.

If you run a titiler tileserver and use a COG file. Not all tiles are created while loading them via a leaflet client.
And I have the same behaviour within my script that asks for all tiles and stores them locally.

202108300645.zip

[meteoDaniel]

The only logs I have :

/usr/local/lib/python3.8/site-packages/rasterio/io.py:133: NotGeoreferencedWarning: Dataset has no geotransform set. The identity matrix may be returned.
  return writer(mempath, 'w+', driver=driver, width=width,
/usr/local/lib/python3.8/site-packages/rio_tiler/reader.py:72: AlphaBandWarning: Alpha band was removed from the output data array

This is I/O stuff, I do not think that this is the problem because the tiles were created but not completly.