Performance Degradation Issue after Continuous Tile Loading in Earth Viewer

[hasanshamsijeze]

I’ve encountered a problem in my application where the globe, map tiles, and elevation data are displayed. I placed a fixed point on the Earth’s surface and rotate the camera around that point using a timer so that the entire globe becomes visible through rotation. Naturally, as new regions come into view, new tiles are loaded.

I used to perform this process before, and after 10-13 hours the application would crash — as I mentioned in issues #194. Now, the app no longer crashes, but after two days of running continuously, its performance drops drastically and becomes very slow.

While the app was running, I went into the settings and disabled elevation tile rendering. The application immediately returned to its normal speed. When I re‑enabled tile rendering, the performance remained good — until roughly another 24 hours passed, after which the slowdown occurred again.

I’m not sure if I need to collect any specific logs or data, but this is the issue I’ve been facing.

[derkreature]

I should wait for @gwaldron to comment.
However, my first thought is that you have exceeded your physical memory and you are swapping to disk.
This sounds like the tile cache has exceeded physical memory and requires a disk cache.
Have you tried logging the number of entries in the disk cache and summing the size of the elevation tiles?

I'm planning on running a memory survey in the next few weeks. I haven't dug into the tile memory pattern used by Rocky yet. I should also mention that implementing a local cache for tiles is prohibited by some licenses.

I'm very interested to hear what @gwaldron has to say.

[hasanshamsijeze]

Our usage is legal according to the terms and conditions provided by the tile provider.

The tiles are stored locally, both elevation and imagery, and the setup is exactly like the at_sample, with the only difference being that the tiles are loaded from a local source instead of a server.

A timer has been implemented that rotates the camera around a fixed point — just like a person standing on the ground and slowly turning around in place.

When elevation tiles were enabled, the system memory usage increased from about 1 GB to 4 GB over the course of one day, and the application became extremely slow. I didn’t analyze any logs at the time, but when I disabled the elevation data, the program immediately returned to normal speed. After enabling it again, the issue reappeared.

For testing, I completely disabled elevation tiles, and after three days the application became only slightly slower than normal and memory usage did not change — but this was not even comparable to the slowdown seen when elevation tiles were active.

However, the issue still exists, and I don’t know whether there is any difference between loading tiles from a server versus loading them locally.

[derkreature]

Awesome. Have you written a trading memory broker over cache?
I don't think that there is a leak, I think that the cache needs to evict over last access time.

If you haven't "touched"/used a tile in the cache over a time delta, evict the tile from physical memory.
This is pretty much what I do for renderers and compositing.

@gwaldron, what do you think?

[gwaldron]

It does sound like a slow leak to me.
Which layer type are you using to read elevation data? TMS? GDAL?

[hasanshamsijeze]

{
"map" : {
"name" : "local map",
"layers": [
{
"name": "OpenStreetMap local Imagery",
"profile": "spherical-mercator",
"uri": "/media/sw/map/map/{z}/{x}/{y}.jpeg",
"type": "TMSImage"
},
{
"name": "OpenStreetMap Elevation",
"profile": "spherical-mercator",
"type": "TMSElevation",
"uri": "/media/sw/map/elevation_tiles/{z}/{x}/{y}.png",
"encoding": "terrarium"
}
]
},
"profile" : "global-qsc"
}
i use of json file and here write TMSElevation so i think use of TMS

[hasanshamsijeze]

I’m loading information from a JSON file and I haven’t set any specific configuration regarding the elevation data type.

{
    "map" : {
        "name" : "local map",
        "layers": [
            {
                "name": "OpenStreetMap local Imagery",
                "profile": "spherical-mercator",
                "uri": "/media/sw/map/map/{z}/{x}/{y}.jpeg",
                "type": "TMSImage"
            },
            {
                "name": "OpenStreetMap Elevation",
                "profile": "spherical-mercator",
                "type": "TMSElevation",
                "uri": "/media/sw/map/elevation_tiles/{z}/{x}/{y}.png",
                "encoding": "terrarium"
            }
        ]
    },
    "profile" : "global-qsc"
}

Also, I don’t think the slowdown issue is related to memory usage — maybe these two topics should be discussed separately.
In this discussion, what matters to me is the slowdown itself, because when I disable elevation data, the program’s performance returns to normal.

[hasanshamsijeze]

@gwaldron please check this issu or help me to handel it

[emminizer]

Okay -- if Linux and Intel then it's almost certainly not the same issue I'm seeing (which appears to be driver related). Sorry for the noise in that case.

[gwaldron]

I managed to run over the weekend with the "spin" feature enabled. On NVIDIA/Windows I did not experience a notable memory increase or any performance slowdown.

Can you post the output of vulkaninfo --summary ?

[hasanshamsijeze]

Thank you.
Could you please provide the version of the program that has the spin feature enabled so I can compare it with my code? Maybe I made a mistake somewhere.

I would also appreciate some guidance on where I should start investigating this issue. Which classes would you recommend I add logging to, so I can find clues about the source of the problem in the application?

[gwaldron]

The "Spin" checkbox in in rocky_demo under the Camera panel.

Your problem may or may not be in the application at all. You can try valgrind to look for memory leaks, or you can try building with AddressSanitizer. I already mentioned searching for a Linux sampling profiler. That's about all I have to recommend.

[hasanshamsijeze]

I looked at your example. In my application, I created a custom MapManipulator. My rotation behavior works in a way that the camera coordinates always remain fixed — for example at lat 20, long 32, alt 20.

My view direction has an angle relative to north, which is initially 0, and by changing this angle the camera rotates in place.

For example, assume the camera position is lat 20, long 32, alt 20 with a 90-degree tilt, meaning it is looking at the horizon. Now, when entering the spin mode, the camera position does not change — like a person standing still and turning around themselves. Because of this, in every view new tiles are loaded.

However, your spin works differently: it keeps a specific point fixed and rotates the camera in such a way that this point always remains in view. Therefore, the tiles do not change.

No matter how much I tried, I could not reproduce my desired behavior using rocky::MapManipulator.