Tagging JPEG creates xmp-sidecar

[oops66]

... The option presumably must apply to all image files, or none, and the wording should be clear about this.

...Right ... I am not an expert in English

[m.Th.]

Well, here is a problem. Theoretically you're right. But in practice, we cannot distinguish between image data change and embedded metadata change...

I don't see a major problem at all - if dates are newer then simply check if any mapped item is different between file and DB. There is not a lot of data to check. If the user has selected to keep embedded metadata in sync then update the DB. If the user has selected to ignore embedded metadata then ignore it.

Really? Let's see...

Let's say that we are in a multi-user/program environment. Let us consider that there are programs which write in embedded metadata (E1, E2, ..., En) and programs which write in sidecars (S1, S2, ..., Sn).

All the following events are for the following image file:



On 1/1/2014: - XnViewMP scans the file and puts in DB the thumb, metadata (if finds) etc.
2/1/2014: The user with E1 enters the categories "Portrait", "John & Vangelis"
3/1/2014: The user with S1, more knowledgeable, fixes the categories changing the "John & Vangelis" to "Jon and Vangelis" (removes the 'h') and adds 'Jon Anderson', ' Vangelis Papathanassiou', 'singer', 'musician', 'electronic', 'progressive rock' etc.
4/1/2014: Another user with E2 (or even E1) makes some edits to the scanned photo (brightness, vibrance, removes noise etc.) but it doesn't touch the metadata.

5/1/2014: The XnViewMP goes again and rescans the photo and what finds?

Finds a JPG with the mdate of 4/1 and a sidecar with 3/1. XnViewMP has the DB date entry for this "double star" 1/1. According to what you're saying, XnView MP will loose the edit and additions from 3/1. Which is bad.

That's why IMHO we need to have like this (scroll to view): (I hope that I didn't left out any combination - also it shows why the sidecar is the preferred method to work with from a programmer's perspective).

Code: Select all

bSidecar := FileExists(Sidecar); //test for the existence of sidecar 

if Photo.mDate > DB.mDate then
{
	if bSidecar then
	{
		if Sidecar.mDate > DB.mDate then
		{
			case AskTheUser of //ok, switch in C++
				0: Import_From_Sidecar; //default
				1: Import_From_Embbeded;
				2: //do nothing - this is the "Cancel" button
			end; // case
		}
		else
			Import_From_Embedded;
	}
	else
		Import_From_Embedded;
}
else
{
	if bSidecar then
	{
		if Sidecar.mDate > DB.mDate then
		{
		Import_From_Sidecar;
		}
	} //if bSidecar
}

[CameronD]

Well, here is a problem. Theoretically you're right. But in practice, we cannot distinguish between image data change and embedded metadata change...

I don't see a major problem at all - if dates are newer then simply check if any mapped item is different between file and DB. There is not a lot of data to check. If the user has selected to keep embedded metadata in sync then update the DB. If the user has selected to ignore embedded metadata then ignore it.

Really? Let's see...

My original comment was in the case that:

If XnView can detect that the image file has been recently changed, but the sidecar has not been modified...

This is not the situation of your example, where both image and sidecar have been modified.

So let's take a few steps backwards...
Possibilities:

1. Image file has been updated after date in DB, but sidecar has not (or does not exist). In this case there should be no ambiguity that any changes in the image file are newer
2. sidecar is newer than DB but image file is not - again there should be no ambiguity in XnView. Even if a user has selected to not use sidecars, XnView should check to see if there is something of relevance in there.
3. If both image and sidecar have been updated since the DB then is gets more complicated. My point is that the sidecar should not automatically be taken as authoritative, and I offer a modification of your example to show why.

On 1/1/2014: - XnViewMP scans the file and puts in DB the thumb, metadata (if finds) etc.
2/1/2014: The user with S1 enters the categories "Portrait", "John & Vangelis" and edits the caption.
3/1/2014: The user with E1, more knowledgeable, fixes the categories changing the "John & Vangelis" to "Jon and Vangelis" (removes the 'h') and adds 'Jon Anderson', ' Vangelis Papathanassiou', 'singer', 'musician', 'electronic', 'progressive rock' etc.
4/1/2014: Another user with E2 (or even E1) makes some edits to the scanned photo (brightness, vibrance, removes noise etc.) but it doesn't touch the metadata.

Note: all I have done here is change who used S1 and E1, plus I added an extra level of change.
If both files are newer than the DB, then XnView cannot make any assumptions at all.
In this example the latest caption is in the sidecar while the latest categories are embedded in the file.

The only sensible course now is for XnView to scan each metadata field (where there are equivalent values in the DB and in the files) and compare against the DB contents.
In the case of the caption, the only changed value is in the image file, and therefore it can automatically be assumed to be newer.
In the case of the categories, both sidecar and file are different from what the DB contains and so XnView should pop up a window showing the two conflicting values and ask which is correct.

Of course, this should be ringing alarm bells with the user, no matter whether they prefer to use embedded data only, or to use sidecars only. Unfortunately this incompatibility between programs has happened to me time and again; I think it is vital that the cause gets identified and the user can then decide some way to work around it.

The question that is still to be answered is: what happens to the embedded metadata that is out of date if XnView has been told not to touch the image file? To my mind, the worst option would be to leave the bad data in place. To answer that question, maybe we need somebody like Oops66, for whom it is more of a concern.

[oops66]

... The question that is still to be answered is: what happens to the embedded metadata that is out of date if XnView has been told not to touch the image file? To my mind, the worst option would be to leave the bad data in place. To answer that question, maybe we need somebody like Oops66, for whom it is more of a concern.

Hello,
For me , in this case it's not a problem at all because the only goal of this option is to preserve (via a single & secure "tick") the originals users files (raw or jpg, etc... files) at the beginning (and by default) ... so modifications (tags, labels,colors,dates,keywords etc..) are only made into XnView DB (or/and sidecar files ,ex: xmp, mie files).
... So in case of customization, it's possible to untick this option, make the modifications to some files, ...and restore the "preserve my originals" option after. (but usually originals raw or jpg files have the right embedded data, dates, etc... by default basically)

[hic]

I would like to sum up this thread a little bit because it get's confusing.

1. The initial request was to prevent XMP-sidecars from being written for JPG and TIF. I guess this could be done by a simple "if" statement before the writing process.
2. The second request was to have an option to automatically update files and XMP-sidecars when database was changed.
3. The third request was, if it would be possible to handle other sidecars in file manamegent operations. These sidecars don't contain metadata, it thus would not be necessary to parse them but only to give an option to consider them in file management, i.e. copying, deleting etc. If in the options the opportunity would be given to define a custom sidecar which afterwards is defined by the system as belonging to image files, it would be sufficient.

Regards,
hic

[m.Th.]

I would like to sum up this thread a little bit because it get's confusing.

1. The initial request was to prevent XMP-sidecars from being written for JPG and TIF. I guess this could be done by a simple "if" statement before the writing process.
2. The second request was to have an option to automatically update files and XMP-sidecars when database was changed.
3. The third request was, if it would be possible to handle other sidecars in file manamegent operations. These sidecars don't contain metadata, it thus would not be necessary to parse them but only to give an option to consider them in file management, i.e. copying, deleting etc. If in the options the opportunity would be given to define a custom sidecar which afterwards is defined by the system as belonging to image files, it would be sufficient.

Regards,
hic

You beat me on it.
I wanted to do exactly the same thing: To sum up.

However let me try to add some value to your post:

1. Metadata placement

We need two options: embedded and sidecar regardless of file type.
I think that this is solved & summed up in Pierre's post here:

Currently if export is enabled, jpeg/png files are updated, and not for other
But if i add sidecar support, they will be

So like that?

Export to:
* None
* Metadata when possible
* Metadata and sidecar
* Sidecar

The Research part is IMHO solved. Needs implementation.

2.a. Automatic Export of metadata

Works at time of writing (if the according checkboxes/options are enabled) but only for embedded metadata.

Pierre just needs to implement the sidecar support here.

The Research part is IMHO solved. Needs implementation.

2.b. "Automatic" Import of metadata

Foreword: Look carefully at the pseudo-Code (NOT at text) in this post

In fact, as depicted above, at this point we already agreed about almost all cases except about how XnView MP will handle the Pop-up Window to the User when both image and sidecar have newer mDate than the date in DB. (I have some comments here, but I want to keep this post clear - comments after)

Research part under development.

3. Custom sidecar support

I think that this post sums up what (IMHO) it can be done with ease.

The Research part is IMHO solved. Needs implementation.


PS: Yes, I know, no SilkyPix/Capture One support. But for this I think that's better to introduce a separate feature: Before/AfterFileOperation events: The ability to execute a parametrized custom script with {Folder} as argument Before/After each CMDR action. Hic (you're the most interested in this) what do you say? Perhaps is better to start another thread for this?

[m.Th.]

Now commenting on the battlefield:

If both files are newer than the DB, then XnView cannot make any assumptions at all.

Agree. Hence XnView must ask the user.

The only sensible course now is for XnView to scan each metadata field (where there are equivalent values in the DB and in the files) and compare against the DB contents.

Here is the problem. While theoretically I agree, practically I think that the process can be rather slow (remember that usually there bunch of files/folders imported not just one or two) and troublesome from user's POV. Asking for each conflicting metadata field for each one of 100 - 600 - 1000 photos is quite a challenge for the user. Also, the code can become complicated.

There are some solutions here to consider:

Solution 1. I think that the only real solution is to ask just once: "Conflicts detected. What to take precedence? {Embedded} {Sidecar} {Cancel Import of metadata}"

Solution 2. Make a list of conflicts (which will grow up very quickly) and show it... ...when? in order to choose for each one metadata field which will be the imported data. Then it needs a bunch of columns: file full path, metadata field name, embedded value, sidecar value, keep XnView value. ...and the user must choose between the three. hmmm....To complicated imho. I would like to have it, but the solution 1 above is mandatory because is the most common case when one enters in a folder with newer metadata.

[hic]

1. Metadata placement

We need two options: embedded and sidecar regardless of file type.
I think that this is solved & summed up in Pierre's post here:

Currently if export is enabled, jpeg/png files are updated, and not for other
But if i add sidecar support, they will be

So like that?

Export to:
* None
* Metadata when possible
* Metadata and sidecar
* Sidecar

The Research part is IMHO solved. Needs implementation.

PS: Yes, I know, no SilkyPix/Capture One support. But for this I think that's better to introduce a separate feature: Before/AfterFileOperation events: The ability to execute a parametrized custom script with {Folder} as argument Before/After each CMDR action. Hic (you're the most interested in this) what do you say? Perhaps is better to start another thread for this?

I agree with you in the list above if in topic two a XMP is created/updated only if wrtiting of metadata is not possible (for raw files). Otherwise we would end up with the current situation that we would have to tick option three and sidecars would be written for JPG, which is useless as no other program recognizes sidecars for JPG.

SilkyPix/CaptureOne sidecars are of minor importance to me, but it could be solved by letting the user specfify not only the custom sidecar but also the subfolder in which it exists.

hic

[CameronD]

Now commenting on the battlefield:

If both files are newer than the DB, then XnView cannot make any assumptions at all.

Agree. Hence XnView must ask the user.

well, I have already disagreed with myself here - automatic processing can still be possible in some cases, as in the following ...

The only sensible course now is for XnView to scan each metadata field (where there are equivalent values in the DB and in the files) and compare against the DB contents.

Here is the problem. While theoretically I agree, practically I think that the process can be rather slow (remember that usually there bunch of files/folders imported not just one or two) and troublesome from user's POV. Asking for each conflicting metadata field for each one of 100 - 600 - 1000 photos is quite a challenge for the user. Also, the code can become complicated.

I think is is much less cpu effort than you think. In the case of embedded metadata , XnViewMP already scans every newer file and automatically updates metadata in the DB. To do this it has already opened the file, read the header and then scanned through it locating the relevant bits of metadata. A string compare is a trivial addition by comparison.

There are some solutions here to consider:

Solution 1. I think that the only real solution is to ask just once: "Conflicts detected. What to take precedence? {Embedded} {Sidecar} {Cancel Import of metadata}"

Solution 2. Make a list of conflicts (which will grow up very quickly) and show it... ..

I agree solution 2 is probably unworkable in its full scope, but solution 1 is next to useless in most situations I am envisaging.

To get back to overview mode: the second part of this thread is concerned with setting out the user's desired workflow for handling metadata.

I regard these sorts of situations that you and I are discussing as anomalous - they are not everyday workflow. I think it would be very rare that hundreds of files are affected in a way that cannot automatically be resolved.

I image the situation could arise in a number of ways:

1. where (new) software behaves in a way that is not expected (or it has a bug),
2. another user operates in a way that conflicts with your chosen workflow,
3. other software cannot be made to operate in a way you desire.

The problem with solution 1 is that it only will work in the 3rd situation.
In the other cases, especially the 2nd, it is most likely that the person facing the choice does not know what the correct answer should be.

Here's a suggestion to an extension to options for solution 1...

There are <insert number here> images with ambiguous/conflicting metadata:

• Automatically assign precedence to sidecar/embedded (this covers the 3rd situation where the user knows what the issue is)
• Automatically resolve as much as possible by scanning and comparing (as I suggested) and return to this window.
• manually review each conflict and allow me to resolve them one at a time. (this is for the user who has no idea what has gone wrong)

I think automatic scan/compare/resolve should be done first when there are fewer than <n> uncertain files in a folder. If <n> is a settable option then user can choose 0 to 1000 according to preferences.

I do not really care how crudely the last option is done. After all, I expect it will be a rare event, but one that comes as a surprise to the user.
One possibility might be in browser mode - highlight any files with unresolvable date conflicts, and highlight fields that do not compare equal when the file is selected.
After all, XnView always has to read the file and metadata to display a lot of the contents that are not in the DB.

[m.Th.]

I think is is much less cpu effort than you think. In the case of embedded metadata , XnViewMP already scans every newer file and automatically updates metadata in the DB. To do this it has already opened the file, read the header and then scanned through it locating the relevant bits of metadata. A string compare is a trivial addition by comparison.

Hmmm... you forget that now XnView MP blindly updates the metadata. Hence, if your proposal will be accepted, (and I'm not against to it - providing that this "field-by-field" mode will occur just in rare occasions) the algorithm will be like this:

1. read the metadata from file in struct S1
2. read the metadata from DB and unpack it (the metadata is compressed in DB) in S2
3. scan structure S1 and for each field in S1 do like this:
___3.a. search the corresponding field in S2 and if found, ask the user and change the value. If not, just add the new value.
4. Write back to DB the S2

This is slower than what we have now - I highlighted the new steps which are time consuming. As I said, of course, this will be a problem when we have many photos (IOW an entire card has "wrong" metadata WRT to our workflow).

Automatically assign precedence to sidecar/embedded (this covers the 3rd situation where the user knows what the issue is)

Well, this is exactly what I said:

...to ask just once: "Conflicts detected. What to take precedence? {Embedded} {Sidecar} {Cancel Import of metadata}"

...After all, XnView always has to read the file and metadata to display a lot of the contents that are not in the DB.

Not quite. Everything is in the DB. All the metadata, file properties and thumbnail. The file is read (scanned) only at the first time - when the thumbs are generated.

[CameronD]

Hmmm... you forget that now XnView MP blindly updates the metadata. Hence, if your proposal will be accepted, (and I'm not against to it - providing that this "field-by-field" mode will occur just in rare occasions)...

I may be getting old, but I have not yet forgotten that. In the current form of the program, there should only be one source of external metadata, and therefore blindly updating it should be valid, so long as the user has selected to keep DB and external metadata in sync.

This is slower than what we have now ... As I said, of course, this will be a problem when we have many photos (IOW an entire card has "wrong" metadata WRT to our workflow).

In the situations I am thinking of, it is clear that something has potentially gone horribly wrong, and if you are thinking of hundreds of images then we may assume it took days of work to get to the situation. Even if it takes 10 seconds per image, that is certain to be far less than I or somebody else spent on the changes. I certainly hope it would be a rare situation.

Automatically assign precedence to sidecar/embedded (this covers the 3rd situation where the user knows what the issue is)

Well, this is exactly what I said: ...

yes, I was just offering reasons why it is not enough on its own.

...After all, XnView always has to read the file and metadata to display a lot of the contents that are not in the DB.

Not quite. Everything is in the DB. All the metadata, file properties and thumbnail. The file is read (scanned) only at the first time - when the thumbs are generated.

OK, maybe I am getting a bit old - I had forgotten about the "meta" blob, but my excuse is I've never been able to tell what is in it . Nevertheless, if the jpeg (for example) file has not changed its modification date then XnView will not have to do anything. If the date has changed, but on only a single data source, then XnViewMP behaves as it already does.

For my part, I have no intention of using sidecars for image formats that can embed the metadata in the image file, so I hope I never see this problem. But I would not bet my house on it - just about every image processing program I have used has done something that I consider a bit whacky, and bad for my mental health.

[m.Th.]

OK, maybe I am getting a bit old...

Heh! Two elders chattin' in a forum. ...perhaps we'll met in paradise in few days from now.

In the situations I am thinking of, it is clear that something has potentially gone horribly wrong, and if you are thinking of hundreds of images then we may assume it took days of work to get to the situation...

Oh, son... You're old, but perhaps I'm older than you. At least in certain areas.

Yes, I have the experience of Joe Clueless doing a Select All on a bunch of important photo subfolders, press Sync Metadata from Catalog and in 10 secs we went in deep, deep mud.
Yes, I know, the backup was the solution.

Even if it takes 10 seconds per image, that is certain to be far less than I or somebody else spent on the changes. I certainly hope it would be a rare situation.

Sure is a rare situation (a corner case) - and that's why I don't press to much for processing speed - but one can reach there very quickly. You know, the mess which one clueless can do in 10 secs, 10 wise men aren't able to clean up in 10 days.

Anyway, I think that we settled down with this long, long thread. (Perhaps the longest - we broke a record!) Let's hope that Pierre will do the easiest part: the implementation.

[Gwenael Q.]

Hello,

A duplicate of this request : viewtopic.php?f=95&t=41270&p=180168#p180168

Thanks for listening