I think, after a very quick look at the Rotate and Automatic deskew actions, the answer is that the XnConvert Automatic deskew action currently does not have the 'Smooth' option available in the Rotate option...
And for completeness, in XnView MP the rotate tool has a 'Use anti-aliasing' option, which may or may not use the same algorithm as XnConvert's Rotate 'Smooth' option, but the Rotate -- Automatic Deskew tool also does not have an anti-aliasing option.
I have attached my own test image, a rectangle aligned with the horizontal axis, which can be used to see the effects of rotation and anti-aliasing when starting with a perfect rectangle. As in your test image, the canvas pixel dimensions are quite small, which results in the image quality degrading more rapidly than would be the case with larger pixel dimensions.
Image 1.png
You do not have the required permissions to view the files attached to this post.