Multiple issues with XnConvert

[dexter86]

1. Turning on "Delete original" makes XnConvert window force focus on itself.
This makes using Windows near impossible. No matter what window you click on, it will immediately loose focus. If you then press Esc on keyboard, it will show you some notification about "aborting transfers"

Clipboard04.png

for maybe one second and then XnConvert will close. My guess is that you are using some Explorer API to remove files, but you are using it in wrong way. And it doesn't matter if I have Recycle Bin enabled or disabled in Windows, or if I set "useRecycleBin=false" in ini, it still behaves the same.

2. Turning on "Delete original" removes original file before converted one is written to disk.
This was a shock to me. I was using Process Monitor to analyze that 1st point and then I saw this

Clipboard03.png

This was a test to convert web to png. First the whole source file is read to memory (not shown here), then it clearly shows that webp file is accessed (CreateFile operation) with intent to remove it (Desired Access: Delete, Delete On Close). After that file is closed (CloseFile operation) so Windows immediately removes that file. After this you can see creating new png file and writing to it. This is so wrong! You can't remove source file before writing destination file, what if there was some error during write operation (bad sector, device removal, etc)? You loose the source file.

3. Portable version stores ini in AppData\Roaming\XnConvert\xnconvert.ini.
Not very portable, is it?

4. Pressing Esc key closes XnConvert regardless of what it is doing.
It will cancel conversion without any confirmation!

5. There is no information that it doesn't support converting gif to animated webp.
It just converts first frame.

6. Output log should display errors at the end.
I converted 40000 files and had 1 error. Good luck finding it.

7. Add "View as: simple list".
I use "Details" view. When I added 40000 files it took about 1 minute to load them to the table with 1 thread, but after they were loaded it started using 2 threads for about 4 minutes with ram usage slowly rising to 1.5 GB to load thumbs and details. Can you add "simple" list view so it will not load details and thumbnails when adding files?

8. When saving files to disk they are written in "4 KB" chunks.
You can see this in that screenshot above, Write operations write to that file in 4096 bytes chunks. This isn't really a very big issue since most of the time we are limited by cpu speed, but writing files in small chunks is just not a good practice. You should use at least 128 KB.

[dexter86]

9. "Export for NConvert" is "badly done".
I tried it just to test if this will get rid of that "delete original" focus problem, but the created bat file was, welll, not properly done.
a) you should be using cmd extension instead of bat (not that it probably really matters, but bat was used in DOS times https://stackoverflow.com/q/148968).

b) When I used "export filenames" it added every filename as parameters to exe. With very large number of files this will result in broken bat file. Nconvert supports filelist from txt file, why don't you use that?

In Command Prompt, the total length of the following command line can't contain more than 8191 characters

https://learn.microsoft.com/en-us/troub ... limitation

c) I changed "Use multiple cores", "Delete original", "Preserve metadata", "Rotate images according to EXIF", and they weren't exported to bat file at all. Probably lots of options don't get exported but I'm not willing to check them.

[xnview]

1. Turning on "Delete original" makes XnConvert window force focus on itself.
This makes using Windows near impossible. No matter what window you click on, it will immediately loose focus. If you then press Esc on keyboard, it will show you some notification about "aborting transfers" Clipboard04.pngfor maybe one second and then XnConvert will close. My guess is that you are using some Explorer API to remove files, but you are using it in wrong way. And it doesn't matter if I have Recycle Bin enabled or disabled in Windows, or if I set "useRecycleBin=false" in ini, it still behaves the same.

It's a known issue when using recycle bin, problem with Windows Explorer API, i can't reproduce when useRecycleBin=false (in this case an unlink is made directly)

2. Turning on "Delete original" removes original file before converted one is written to disk.
This was a shock to me. I was using Process Monitor to analyze that 1st point and then I saw this

yes, need to be changed when 'delete original' is on

3. Portable version stores ini in AppData\Roaming\XnConvert\xnconvert.ini.
Not very portable, is it?

xnconvert.ini need to be in same folder as XnConvert

4. Pressing Esc key closes XnConvert regardless of what it is doing.
It will cancel conversion without any confirmation!

you means during processing, need to show a confirmation dialog?

5. There is no information that it doesn't support converting gif to animated webp.
It just converts first frame.

Animated format is not supported, so only first frame is loaded

6. Output log should display errors at the end.
I converted 40000 files and had 1 error. Good luck finding it.

Perhaps copying all errors at the end of logs?

7. Add "View as: simple list".
I use "Details" view. When I added 40000 files it took about 1 minute to load them to the table with 1 thread, but after they were loaded it started using 2 threads for about 4 minutes with ram usage slowly rising to 1.5 GB to load thumbs and details. Can you add "simple" list view so it will not load details and thumbnails when adding files?

ok, i'll add it

8. When saving files to disk they are written in "4 KB" chunks.
You can see this in that screenshot above, Write operations write to that file in 4096 bytes chunks. This isn't really a very big issue since most of the time we are limited by cpu speed, but writing files in small chunks is just not a good practice. You should use at least 128 KB.

Saving in which format?

9. "Export for NConvert" is "badly done".
I tried it just to test if this will get rid of that "delete original" focus problem, but the created bat file was, welll, not properly done.
a) you should be using cmd extension instead of bat (not that it probably really matters, but bat was used in DOS times https://stackoverflow.com/q/148968).

ok

b) When I used "export filenames" it added every filename as parameters to exe. With very large number of files this will result in broken bat file. Nconvert supports filelist from txt file, why don't you use that?

In Command Prompt, the total length of the following command line can't contain more than 8191 characters

yes, will be better

c) I changed "Use multiple cores", "Delete original", "Preserve metadata", "Rotate images according to EXIF", and they weren't exported to bat file at all. Probably lots of options don't get exported but I'm not willing to check them.

Not all options are supported by NConvert

[dexter86]

It's a known issue when using recycle bin, problem with Windows Explorer API, i can't reproduce when useRecycleBin=false (in this case an unlink is made directly)

With useRecycleBin=false it's still the same, I think you may be wrong and have a small bug where you don't call unlink. Here's my ini if you want to check
https://mega.nz/file/HpNnTQrZ#98f8IjP2H ... DSRr_CIZNg
and here's a video recording of this
https://mega.nz/file/LtklhYDI#9PdswvzC5 ... wfZMebihD4
At the end of the video I press Esc and you can see for a split second that dialog I wrote about before. And I can add another thing, it doesn't close XnConvert, it throws an exception

Clipboard06.png

xnconvert.ini need to be in same folder as XnConvert

Ok thanks, I moved it.

you means during processing, need to show a confirmation dialog?

YES, and pause processing when that dialog is shown

Animated format is not supported, so only first frame is loaded

But there is NO INFORMATION about it! Even on https://www.xnview.com/en/image_formats/ or somewhere in XnConvert. There should be even a warning when user adds animated gif files!

Perhaps copying all errors at the end of logs?

Or have two different logs, one for "Information" and one for "Errors", either side by side, or in two different tabs.

Saving in which format?

JPG and PNG save file in 4KB chunks, but WEBP doesn't, it saves file in few smaller chunks and then one big chunk. I only tested these 3 formats, here are some screenshots from Process Monitor:

Clipboard02.png

Clipboard04.png

And let me add another thing, (although I mentioned this previously in no. 7). When you add files to XnView, they are read twice, first when they are added to listbox, and second time when exif & thumbnail information is read. Process Monitor clearly shows this. And to make matters worse, JPG and PNG are read in 4KB chunks, but WEBP is read in few smaller chunks and one big chunk, so it looks like every format behaves differently. I hope that when you add "Simple list" view, you code it properly and file will be read once and if one switches to different view then xnconvert will load exif & thumbnails.

Not all options are supported by NConvert

Then show some warning before export that not every option is supported by NConvert.

[xnview]

JPG and PNG save file in 4KB chunks, but WEBP doesn't, it saves file in few smaller chunks and then one big chunk. I only tested these 3 formats, here are some screenshots from Process Monitor:

It depends on formats.... Why it's a problem?

Thanks to your detailed description I can reproduce RecycleBin issue,
List mode,
ESC issue,
Source file deleted issue

This makes using Windows near impossible. No matter what window you click on, it will immediately loose focus

I have not this issue on Win10, focus will never change

[dexter86]

It depends on formats.... Why it's a problem?

Reading and writing files in small chunks is much slower than when you do this in bigger chunks, even when all files are defragmented, doesn't matter if it's hdd or sdd. And it becomes even worse for NAS over standard SMB protocol (TCP overhead and SMB protocol overhead)

ssd.png

hdd.png

I have not this issue on Win10, focus will never change

Try on another PC, or install VirtualBox and in it a fresh Windows 10/11. Or you can build me some debug version with extensive logging and I will test it for you.

[xnview]

These problems are supposed to be fixed in XnView MP 1.5.4. Please check and confirm the bug fix here.

[dexter86]

Sorry for late reply

Ad 1. Not fixed, I told you to make me some debug build. Just add lots of debug logging to that part of your code and let me test it
Ad 2. Confirmed fixed, but there is one small issue, just after writing the resulting file, xnconvert opens and closes source file multiple times before it removes it (You can see this in Process Monitor). It adds very small delay to overall conversion process, but it's negligible.
Ad 3. XnViewMP portable asks where to store ini and other files on first run. XnConvert should do the same.
Ad 4. Not fixed, XnView just crashes, i sent you a crash report
Ad 5. Still no information ANYWHERE that it doesn't support converting animated gif to animated webp
Ad 6. I didn't check if you changed it or not.
Ad 7. Confirmed fixed, "List" view works as intended
Ad 8. Still not fixed, jpg and png files are read & written in 4KB chunks
Ad 9. You didn't do anything about it, still uses bat extension, still adds every file as parameter to exe instead of txt filelist. Adding veyr large number of images to list the resulting bat file doesn't run, it closes right after you open it. If you try to run it directly form cmd you will get "The system cannot execute the specified program.". Also still doesn't export most selected option, for example "Delete original" is supported by nconvert but you don't export it to bat.

[xnview]

Ad 1. Not fixed, I told you to make me some debug build. Just add lots of debug logging to that part of your code and let me test it

I can't add more debug, i call a win32 api, that's all. But it's strange, because in 1.6.1, i've changed the method to unlink (without changing focus)

Ad 3. XnViewMP portable asks where to store ini and other files on first run. XnConvert should do the same.

right

Ad 4. Not fixed, XnView just crashes, i sent you a crash report

I can't reproduce. Do you have actions? do you use multi threading? Which output format?

Ad 8. Still not fixed, jpg and png files are read & written in 4KB chunks

I can't change the way that format write datas, because some use third party library. And i think that we must keep OS to manage that...

Ad 5. Still no information ANYWHERE that it doesn't support converting animated gif to animated webp
Ad 9. You didn't do anything about it, still uses bat extension, still adds every file as parameter to exe instead of txt filelist. Adding veyr large number of images to list the resulting bat file doesn't run, it closes right after you open it. If you try to run it directly form cmd you will get "The system cannot execute the specified program.". Also still doesn't export most selected option, for example "Delete original" is supported by nconvert but you don't export it to bat.

Currently i have not yet changed that...

[dexter86]

I can't add more debug, i call a win32 api, that's all. But it's strange, because in 1.6.1, i've changed the method to unlink (without changing focus)

I tested it with 1.6.0 because it was the latest version at the time I wrote that post. I just tested this with 1.6.1 and it's still the same, see for yourself https://imgur.com/a/9DvK8gr. Can you at least post here that fragment of source code you use for removal? Preferable the whole fragment with if/case.

I can't reproduce. Do you have actions? do you use multi threading? Which output format?

I used the same settings like in that video above. You can use the same settings like in that video and during conversion just switch to any other opened app and hold Esc key. XNConvert doesn't even need to be active, it will still crash.

I can't change the way that format write datas, because some use third party library. And i think that we must keep OS to manage that...

Which library you use for jpg and png?

[xnview]

I can't add more debug, i call a win32 api, that's all. But it's strange, because in 1.6.1, i've changed the method to unlink (without changing focus)

I tested it with 1.6.0 because it was the latest version at the time I wrote that post. I just tested this with 1.6.1 and it's still the same, see for yourself https://imgur.com/a/9DvK8gr. Can you at least post here that fragment of source code you use for removal? Preferable the whole fragment with if/case.

In 1.6.1, i use QFile::moveToTrash (that uses IFileOperation), and if there is a problem i use this code

Code: Select all

		SHFILEOPSTRUCT file_op;
		memset(&file_op, 0, sizeof(file_op));
		file_op.hwnd = (HWND)parent->effectiveWinId();
		file_op.pFrom = lpFrom;
		file_op.fFlags = FOF_ALLOWUNDO | FOF_NOCONFIRMATION; // | FOF_SILENT;
		file_op.wFunc = FO_DELETE;
		bool is_ok = SHFileOperation(&file_op) == 0 && file_op.fAnyOperationsAborted == 0;

If i remember correctly, i've tried FOF_SILENT flag without change for users with the problem... But i can send you a version to test...

I can't reproduce. Do you have actions? do you use multi threading? Which output format?

I used the same settings like in that video above. You can use the same settings like in that video and during conversion just switch to any other opened app and hold Esc key. XNConvert doesn't even need to be active, it will still crash.

Ok, but in XnView 1.6.1, when you press ESC, you should have a warning dialog?

I can't change the way that format write datas, because some use third party library. And i think that we must keep OS to manage that...

Which library you use for jpg and png?

libpng / libjpeg

[dexter86]

In 1.6.1, i use QFile::moveToTrash (that uses IFileOperation), and if there is a problem i use this code

Code: Select all

		SHFILEOPSTRUCT file_op;
		memset(&file_op, 0, sizeof(file_op));
		file_op.hwnd = (HWND)parent->effectiveWinId();
		file_op.pFrom = lpFrom;
		file_op.fFlags = FOF_ALLOWUNDO | FOF_NOCONFIRMATION; // | FOF_SILENT;
		file_op.wFunc = FO_DELETE;
		bool is_ok = SHFileOperation(&file_op) == 0 && file_op.fAnyOperationsAborted == 0;

If i remember correctly, i've tried FOF_SILENT flag without change for users with the problem... But i can send you a version to test...

Nevermind that, it's working fine!
I used 1.6.1 portable with default settings, so it tried to use recycle bin, but I have recycle bin disabled for every partition. And when I enabled recycle bin for that partition it now works ok. And after that, I disabled recycle bin for that partition and changed in ini "useRecycleBin" to false, and it also works fine .

Ok, but in XnView 1.6.1, when you press ESC, you should have a warning dialog?

Esc key now works fine after those fixes I've written above. Previously that weird removal dialog box hijacked focus and that caused a crash when I was holding Esc in another aplication. Now you only need to unify "Esc" behavior and "Stop" button. Both should behave the same and show onfirmation dialog "Cancel batch processing" and when user click on Yes it should just cancel batch, instead of closing whole batch window. And another thing to fix, you should pause batch when that "Cancel batch processing" dialog box is displaying.

[dexter86]

And another thing, since you use ffmpeg, you could support converting animated gif to animated webp
https://superuser.com/a/1723044

Or use gif2webp which is a part of libwebp
https://developers.google.com/speed/webp/docs/gif2webp