Out of memory when recursive viewing 20.000+ videos

[jadO]

It's not the case, when thumbnail is creating (whole folder option on or off), the thumbnail is kept in memory.

If you have 20000 files in thumbnail view, you'll have 20000 thumbnails in memory if 'create thumbnails for whole folder' is on. And when off, the thumbnails in memory will grow following the scroll...

This would explain the excessive use of memory. But it would also be bad design. Why should XnVieMP have 20000+ thumbnails in memory? There is absolutely no reason for that.

Additionally I have just checked a running thumbnailing task. It is running as whole folder and has constantly 800 MB of memory. According to you statement, the memory must grow as it keeps getting more and more thumbnails. But it is not the case. I am having between 800 and 809 MB of memory usage. Sometimes it falls down to 800 some times it is up at 809.
With adding more and more thumbnails to memorey, it would not be possible that memory usage would shrink. I believe.

[xnview]

it's why you have 'create thumbnails for whole folder' setting. If you work with large folder, you need to turn off this setting

[helmut]

it's why you have 'create thumbnails for whole folder' setting. If you work with large folder, you need to turn off this setting

From what I can see the "problem" is setting "Create thumbnails for whole folder" in combination with "Show all files (recursive)". This recursive view on a specific folder is supposed to show all files in the current folder plus all the subfolders in the browser. This means that when using the recursive view you force XnView to display 20.000 files in the browser. All thumbnails shown in the browser at a time must be loaded into memory, so in your case all 20.000 thumbnails.

I don't know what the setting "Create thumbnails for whole folder" currently does. This setting should apply to the current folder, only, and not apply to the folder plus all its subfolders if recursive view is activated. But if I understand right this will not solve the problem that in your case: In recursive view 20.000 thumbnails have to be loaded and displayed at a time.

[jadO]

it's why you have 'create thumbnails for whole folder' setting. If you work with large folder, you need to turn off this setting

From what I can see the "problem" is setting "Create thumbnails for whole folder" in combination with "Show all files (recursive)". This recursive view on a specific folder is supposed to show all files in the current folder plus all the subfolders in the browser. This means that when using the recursive view you force XnView to display 20.000 files in the browser. All thumbnails shown in the browser at a time must be loaded into memory, so in your case all 20.000 thumbnails.

I don't know what the setting "Create thumbnails for whole folder" currently does. This setting should apply to the current folder, only, and not apply to the folder plus all its subfolders if recursive view is activated. But if I understand right this will not solve the problem that in your case: In recursive view 20.000 thumbnails have to be loaded and displayed at a time.

Excellent post. Thank you. But again. What ridiculous use case is it to have 20.000+ thumbnails in memory? There is no such use case ever. So its bad design to even try go load this number in memory.

[helmut]

... Excellent post. Thank you. But again. What ridiculous use case is it to have 20.000+ thumbnails in memory? There is no such use case ever. So its bad design to even try go load this number in memory.

jad0, don't judge a technical design and call it bad because it doesn't support your specific use case. Let's rather try and find a solution.

Standard use case of XnView is browsing and viewing (recursive or not) folders with some hundreds of images. This works very fast and well in XnView because thumbnails are loaded from cache or generated on the fly and cached and stored in memory for viewing. When scrolling the thumbnails area, new thumbnails are shown right away.

- ATM, I think your use case with more than 20.000 video files is very special and unusual. Your use case should be supported if possible but design decisions have been made which might prevent this. Perhaps you can describe your use case in more detail (folder structure, min./max./avg. size of the video files, private/business, ...) and explain why it's also a standard and usual use case and therefore supposed to be supported well by XnView.

- Could you please try and see what happens if setting "Create thumbnails for whole folder" is deactivated? Do you still have problems, then?

- What's the average size of your video files? FAIK, with setting "Show 4 thumbnails in video's thumbnails (instead of 1)" the whole video must be loaded to create these four thumbnails because thumbnails are at specific percentages of the video. With "Show 4 thumbnails in video's thumbnails (instead of 1)" activated your 20.000+ video files require to generate and load 80.000+ thumbnails. Does deactivating "Show 4 thumbnails in video's thumbnails (instead of 1)" help and is this perhaps an option?

[jadO]

Thanks for you post again helmut.
My use case is not unusual. The design decision is what is unusual.
Why is that? Because it definitely works if recursive and entire folder is not switched on.

So everything should be there to build tumbnails without a memory leak.

Here is the problem: Without entire folder on I scroll down in folder -> No memory issue. This means that creating thumbnaily one by one does work as it should.

With entire folder on -> memory issue.

So all I am saying is: Build a temp list of all files that have to be thumbnailed, create a thumbnail one by one, discard thumbnail from memory when done. Like when entire folder is switched off.

[xnview]

but why you don't use 'create thumbnails for whole folder' OFF?

[helmut]

Thanks for you post again helmut.
My use case is not unusual. The design decision is what is unusual.
Why is that? Because it definitely works if recursive and entire folder is not switched on.

I don't know anyone except you who has 20.000 or more video files on his computer. So I still don't see that this is a standard use case. What's is your use case to scroll through 20.000 or more videos?

So everything should be there to build tumbnails without a memory leak.

How do you know that there is a memory leak and not just a building up of memory because of your 20.000 thumbnails (probably it's even 80.000 thumbnails because of your current settings)?

So all I am saying is: Build a temp list of all files that have to be thumbnailed, create a thumbnail one by one, discard thumbnail from memory when done. Like when entire folder is switched off.

Even after repeating several times you don't understand: For viewing a thumbnail, a windows object has to be created that needs memory. Your 20.000 video files generate 20.000 thumbnails (or probably even 80.000) which means 20.000 thumbnails in memory. You cannot release this memory because you need the object for viewing. Understood, now?

jad0, support the search for a solution by following the suggestions that were given to you. Complaining all the time and posting your theories how things should work and should be changed doesn't help in any way. You are free to look for a different graphic viewer who supports your use case and which can display your 20.000 video files quickly. Please let us know if you have found such a graphic viewer which you can use for free.

[jadO]

but why you don't use 'create thumbnails for whole folder' OFF?

Because this means to manually scroll through 20000+ videos to have their thumbnails created. Do you have an idea how much time this would consume?

What's is your use case to scroll through 20.000 or more videos?

helmut, this is required to have thumbnails created initially.

How do you know that there is a memory leak and not just a building up of memory because of your 20.000 thumbnails (probably it's even 80.000 thumbnails because of your current settings)?

You are right, I don't know if there is a memory leak or not. All I can say is: XnViewMP consumes an incredible amount of memory and causes even Windows to malfunction. With that setting switched off, there is no such failure. But when I can scroll manually through this number of files and have the thumbnails created without that failure, it should be possible to do it automatically too.

Even after repeating several times you don't understand: For viewing a thumbnail, a windows object has to be created that needs memory. Your 20.000 video files generate 20.000 thumbnails (or probably even 80.000) which means 20.000 thumbnails in memory. You cannot release this memory because you need the object for viewing. Understood, now?

Why do you think XnViewMP should hold 80000+ thumbnails in memory even when not looking at them? But this is not even the issue because it is not about looking at thumbnails, it is about generation of thumbnails with that particular settings that causes XnViewMP to use an incredible amount of memory which causes Windows to behave erratically. This is not what should happen.

jad0, support the search for a solution by following the suggestions that were given to you. Complaining all the time and posting your theories how things should work and should be changed doesn't help in any way. You are free to look for a different graphic viewer who supports your use case and which can display your 20.000 video files quickly. Please let us know if you have found such a graphic viewer which you can use for free.

Well it was my suggestion to use ffmpeg for thumbnailing which was a big improvement for XnViewMP I believe. Please refrain from telling me what I should complain about.

[helmut]

helmut, this is required to have thumbnails created initially.

So your use case/intention is initial thumbnail creation for 20.000 video files. I assume the thumbnail creation is intended that later you can browse the folders with videos quickly.

For initial thumbnail creation there is Settings >> Browser >> Catalog >> Catalog tab. Use "Add folder..." and select a folder. This will force XnView to create thumbnails for the specified folder and all its subfolders. Thumbnail creation for 20.000 video files will take a while. In the meantime you can start a second instance of XnView and do other things if needed.

[jadO]

So your use case/intention is initial thumbnail creation for 20.000 video files. I assume the thumbnail creation is intended that later you can browse the folders with videos quickly.

For initial thumbnail creation there is Settings >> Browser >> Catalog >> Catalog tab. Use "Add folder..." and select a folder. This will force XnView to create thumbnails for the specified folder and all its subfolders. Thumbnail creation for 20.000 video files will take a while. In the meantime you can start a second instance of XnView and do other things if needed.

And you believe this will not use this excessive amount of memory?
This would prove the issue I have mentioned.

[helmut]

helmut, this is required to have thumbnails created initially.

So your use case/intention is initial thumbnail creation for 20.000 video files. I assume the thumbnail creation is intended that later you can browse the folders with videos quickly.

For initial thumbnail creation there is Settings >> Browser >> Catalog >> Catalog tab. Use "Add folder..." and select a folder. This will force XnView to create thumbnails for the specified folder and all its subfolders. Thumbnail creation for 20.000 video files will take a while. In the meantime you can start a second instance of XnView and do other things if needed.

Did the creation of thumbnails described above work without problems, jadO?

[jadO]

Did the creation of thumbnails described above work without problems, jadO?

Unfortunately not. Examining the functions a lot of strange stuff showed up.

Prerequisites: 4 thumbnails enabled and ffmpeg, all files have been thumbnailed already thumbnailed with non ffmpeg from previous versions.
Now with setting "Create thumbnails for whole folder" off memory footprint starts small. However whith scrolling down it increases. I stopped when it hit >1 Gb.
The same behavior can be seen when turning "Create thumbnails for whole folder" on: Memory increases automatically until > 1Gb and at some point XnViewMP acts weird again

From this behavior it seems to be safe to assume that while creating thumbnails, XnViewMP is loading or trying to load all existing thumbnails into memory. I don't see any reason for this behavior unless they need to be displayed in Explorer.

Second finding: When using the rebuild function via catalog I had a crash during creation and after restarting the application all existing thumbnails from that folder were gone. I chose another folder with existing thumbnails and clicked "Abort" button during recreation only to find that all existing thumbnails from that folder were gone too. It seems that XnViewMP deletes all existing thumbnails before recreation so there are no thumbnails it can try to load. This would explain why the memory footprint remains small using this fucntion.

But this also means that I cannot use it..

Another thing I was missing in the catalog setting is to remove non-existent paths. I was not sure about the "optimize" button. I think I remember that it tends to break everything or removed all categories and tags or something.

So maybe Pierre wanna check if my findings are accurate: 1. Thumbnailing via catalog settings deletes all existing thumbnails from selected folder before thumbnailing starts. In case of crash or aborting all existing thumbnails are lost.
2. In explorer view XnViewMP tries to load all existing thumbnails into memory before or while thumbnailing and probably never releases then which causes memory shortage.

[xnview]

. Thumbnailing via catalog settings deletes all existing thumbnails from selected folder before thumbnailing starts. In case of crash or aborting all existing thumbnails are lost.

yes, i purge thumbnails before...

2. In explorer view XnViewMP tries to load all existing thumbnails into memory before or while thumbnailing and probably never releases then which causes memory shortage.

yes it's normal if you have 'create for whole folder'

[jadO]

. Thumbnailing via catalog settings deletes all existing thumbnails from selected folder before thumbnailing starts. In case of crash or aborting all existing thumbnails are lost.

yes, i purge thumbnails before...

I don't think this is a good idea. In case of abort or crash all thumbnails are lost.

2. In explorer view XnViewMP tries to load all existing thumbnails into memory before or while thumbnailing and probably never releases then which causes memory shortage.

yes it's normal if you have 'create for whole folder'

Why is this normal? I don't see a reason for loading all thumbnails until everything crashes.
Edit: Also my findings showed, that simple scrolling through a large number of thumbnails increases the memory dramatically. Memory management seems to be absent.
There should be strict memory control. I only see 2 reasons to load a thumbnail: When it gets displayed or processed. When it does not get displayed anymore its memory should get freed and after it has been processed it should free its memory too.