Search…

X3 Photo Gallery Support Forums

Search…
 
littlelio
Experienced
Topic Author
Posts: 25
Joined: 20 Nov 2018, 21:21

A particular jpg file breaks the gallery page of a folder

18 Jan 2019, 13:25

I am not sure if it is a bug, but this is just super weird...

I was uploading around 300 photos to a folder, but somehow the page got broken. The 'gallery' admin page of a folder is filled with funny characters, such as:
Code
f� ���g��2���QÇN͌� �q�U�Y_i�~c�L�%�S[�#�.�B�}�kV�����$�'Xft'oA����������ڤ�ɬ�2G���&�@sF�bO�<�s��8�2ھ�\q�>0���迗l﵍.�>Z�R ��$��?��}���8�m��_���^ٹJ9�Y#��������H:uc� �X�4����@���p1y.�p��8�<��X��/����aK�e^v���'g�-sQ��7��;�����"�8������/��A������J����џ�9�KG������$x?�9q����?$y�=N�;}*Y`�1�G�7��L�J�-�mtxe}>�<�mj�3�ˏG������҂򖛭��淧j���V�E��"�u�C�� �>�i;z&"<]�r���$�-�����g����c��W�+� -��W��V���2ܽ:�ѯ����G"[��ߏ�;m<���(L�2���!��o�?��c�WȾb�e�1�^G�h�ȟZY�����N��c�،ī����h�||X���|_���Y�m�S�{��{%���i��I� x�¼V�e��7Z�<�*5qt=����7�����Q�l��Dy����z6��Z�h��ȉ��5����6`���v�O#�F��>��b�/2��pqG���7�X\���h��$���Yت(?l�aA����s���2N����!�s�šy|�o��w�W��\���r�!���h>&�YnM|�*C��pk�,��&Q�'��.�[0�]f���a��;؈ ��5b$��A�?˜��J�����E�&�BCƌ�r�)���藡�����h�Ư�՟QI9�iz}�b�5����Mf�K��O�Ei������v��TP?c.8�����<����̱Y�[;KR���ciU�B���ƃ���;OӚb#o�/�v&�q��c����C���R�4m;�Z\V�� ��,�T�y��mȶ��0�2� ��~�#�e����X����6{�hO��M�����x�/,�Z�V���.�R�R)�ť�� htxyn��g�9F�W,ga���E��(bǖ1���<�r����<_є)�24�W�/����4���B]!m�4b�$-Ӊ �� ��o��k��8���?���_���//���Ey���Tt�t�ⲷ� ��`"�b>�1ʨ�:r�-^�,���>��� z���K����⯦?�|�!$�7/�����#<�Α�p�6�(�o3�¸��ξj���`kXc�����ԧώc��B��&~����p�ɮ��x-��$+$B{H��`�Y�aOUcz�m�|_��9��;��k�-h;���G��������,'\���Ӣc'���'Y��jy���?eJB3��K��3�2Gx���W���̿������!U�DT��n1!pS�&`_�n9���vl��\ٸ��'��?�����r�����^g14%*�[��[��6X%��O1��p��G��|�g��z��q��"�����4�~�������UIϒ_P�̖?]f����YE4���ߎ]�Y!q4����r�c���֧�JI��`��d��s�?�F��T��tc��YɡQ���{;D�g�G�(C����^�/��wqi�9~D cqu%��� %֪o�-���N�)f�rG�e$�c����#�,2+�IJ���e��o<���Қ5�j@|��:�����S�}L�øbT����m����ײ�5���dm�ʭ�*���j��.�kL��5d:���/}h�4!�����CU?v�vЈ�I���a�Ap�C���czՔ�m&X�3\%ږva�%Zq���o���j���l�9Edp������/��Y��v�wR6_8aަ�/��.5{ķ�J���nܨ�3��=H���Ȟ���p���G脽8���Iux/���ڦ��]ډH%�(�+ljR�Fc�߳��#6�"y�⁌O��x�8�N����?���!��S0/,m���A�CT�� ��9~y�@��K����a�a�v��t�t���q� -��eؚ���?�~�i�3R��h��.�c������� �c���hA[����b*��e����i����FP�v�(��)��s�y�aj��j6��PȥM?e����E�0��j���� �3��7���X���k����a-��S$h�%�C¿j���#��>����͓P*���C�%�w���]�Q����*O�k�"�CO��K����[�r��F�b˗���o�3��2��h��X�S9[+9]���%�b�?g'�<�p���0�,����̑�w("/���V�P��(�s�ӌ�( ���@|K�Ɏ2��<���O3D�}_������'���<��j;�$�N2����X�c+�<�Y���"��mͯ��!(�1����o��>?�|����7S^]�d��Y㗒��?�I�=�c��������/�Ꮻ��O���l��'��m"��A��UL*x��r��#3���<�?�;���=UB�+�������~"ǵMB����P�� ���L!Ug)��drWq�9��<}�<�l�<�U ��}_�J/e��7�-����r�RH��*�8����hO��'6���Ջ���!����oԐ~}����[x�El�0D�ȡT�b������ia���s]��J������̚�%��X5Sŀu�l|x�n�㈘�~[����ʄ�WY(|��"c�H���,�rgR���:������/m�a"��&aF�^a[�-�e�/b�sFD@�C�e���^z������k:�Gw,Mp�''�V�F�GSʏ�c�%���^G�n�ka��tז��-��?N �8:�~/����y��`lć����bAb6 ��Ʉj+{�>��>a��u4�V��X^��d�P:����oQs���Lj_.�ϡ�(c�M�e78�L��[�{h⸜�3� '��"�?*|f��U�W��9�\2��m��38�D�p�O����_��P���3�{�V�U����wsp;;h~'���on峷'����yr�1�ї����<��;�uw^��ZM=�J�� 4d*[�]ד�~���#\'�r1��H�Oe���8�y����ǹ��Es$���Z�L�t�F�z���v�C�h�]uKǪ=��f�r�EV�ܣ�1���?H���g��kfnJ����6���|�v���Q�v�L�p��O�6���i赖ѥ��0�ǩ������#�1��)#�1��]H�����82�����p�y(�����O�-R����ϧ�q��$�S�R�c��~ڕ?�s3Q��|@�6��ڸ��һ�'jVz���Y�ae{�HBXB�eCR T�,���~��P:�(��s���_�����ς��G���?��?�q��22��yd���og��-�Ѳ �4�yJN��\�l\D����~$���9�G<��w��?Ќ#(���G�c[�t�aЧӮt3)/l/d,��⤀�V'�m1jc�h�N�Y��G)�s}s��2���?�xT��2y��;��Hlnm�������#jU\^� �$A�������8�8O$#���x�1�w������鷶���*l���3�_��&5��nO2k:m�M$?W�{��(H�d�*ժ�9��T��o�������?�}!������)�DA+]�����f�Cjs��J����1������K��x���!�)~�s-��T*Dr@ڭˮgvx��n���^8c�8����N\����ׯv'��fW��C�w&-��0W��=�������B��]�O��n`ZYƷV�8�t����x~�,����؊oO��8��qN�e�����W�?�"WT���ze����;$\�IPȂ6�y�.����cJ2l5��:���_��o���9�q��I�+��ҝ6ȸ)�/%�MY����WQ���R��4��UɃ�QF�>�������F�<����}6�_*�E%��7���4�Y�K�Lȣ��#W��T������s��k)����Q�F���x?����)��q���f��]B���!���x��8�~�C�!9d�����/?��||��!#��~��F���K���!�O���O�����ʢMJ�ӑ��U�܏��"8�r=?�:���X�a)qzq�\Qɏ�V͏�o%��{}V�/2Fb"��Vjz��P�벲��Y��/?,G��?�j^aҵ_����]6;��"sf`U~%�[�.Ӂd�c��&���Y$7�Rp�J�2vW�A�N��/k�q.wqs{6<@Ǩݒi�zl2��ޢ��*�f��9�-0����I��m�㍞P!�`@�ia�N���Q�Y=����vB� 5�X�U+E��5#�nLH��p�υ��R��[��.�iG��l���'~�~96a䈖nw�Ο���=�)�' �?�?���"��=+YD��P��z�Q��U��pEC���?��`ɋ$%dq���������f!,w�f������_鸸��EZZ��c�MZ�YӶ�I��+�44��_��7+���?_S�r��8OO��a�/�rG��z���5���f�U��d�:��.L�M�Q�~#��?�ȿO�)z����_�ZNI�9�&8�r�f����i߯ݗ�erh��ِy9%ҭa{�౗Pc��p�]�H)����׿���b���������I���˦v`�wl�eYd�B� ;���򳥐�#x�>�S��$I FU��H���&!X��ѡ<�s�:�]��Z}�1�^��3�h�H�5�e�K;Y��[1ZK _QA���\�U�V!r�~"�;K��W����B?\���fM� N��I����t�Vڡ,��~��'�u�VV� f�����0�����B����}_����Ck$�M�n��#%���"��?�����ǎhN�W��'0���9dV?���}L�X4�m;9���]�[ �L��X��CN\[��}4Ɉ?���+�Ny�(�N��~����q�o�XE�h��m�4���:�� �.R;�]۟�+f@���<���u٢e��DqN>$���O������c�yO��O0ji�!3!Vnd�;r�)�f�H��E���RI����(%��=�!�2��t���r���\!�T�ܢ�_k�+��OQCP ��$����[ _r4�U�D��?�5+�X�.!���r�2ٚ�Y���S�9ӷ1�Xn�����)�Ao*�. �0f H�.]rB'�Ш�F�p8H�.8�����C���m�Q���q{qì$P�Z;f!yG^<k��ٙN���0��k�Eo|��kl@Ă������o�?͕vdc9�L�2����C��(h��_����V8�§�_�kF���im����y"�[�;�X�c+�;l�W����&3���tg�F�������{��P�:p��a��!��vx��!B`�hVV�:-����E ¯��\ ���������>��/$�� K�_M��1f�����y.�)�v�iLP��Hc^M�٩n?/�9�8����t�\��*��f��l�����$�f��B�ˆ�Z�m�:�>���� A�G�T�����d\2���2�Er;x̸�Q�Y,\�Q?V?�����ܚ��5�y9!�h�u!��;}��#?݀�<�`Y�Kde؃�f$6��A_�ݭ34�ƨ�)T���q˄{�8�L��|�q���Y��T+KnUz� ��#��#u;ԋЖIB/�Y(8R �~���$>��-�/� �F�qG`)N����T�o8y*o,��vg��mSOI�U�ip(��qE�#-0�����楥K���a{{�y4�y�QJ.��u�iφ`��;�Ɍ<�wR�1�����q��3\E�%��e;�n�j䑴�J棕N�9�����8��������`�Ò�O�O �0���z��hWP\:���8 ��M�<�o�/��������LLG�?��#�?��H�8e����ԭQg��$���:�B�*�t�D�>���=��ŧԈH�_��^Q(�Q� g�]�-N���d��P1��օ%������2�]��� `ۋ�8���?��ڨpN#!�F�^W�'×^�<��p�a^i�ӓ�R=��v��gxd�d�VoU z|b��;L$��]^�����}2��8�?�)��܈5K��omQ�)�;n�d��3[ �~��>����z��P�S�^5,���ݿ���{$�3^����(i���&���?���<�W�&�7����4p�)C ��F���f�S�=�;ͤ���R�������� �y��*|<$W=�cjg;���ιz�0��"������Gms41����W�A١��R�V����:�k��k=G�@�QO���*>�g> q@����zg���4�Q��>j��9�j#/��_����շ���[�aE�4�$èۂ��@����A�� /j�zr���9�t:�Ƅ��G_�g������XO��%4�ι?^0G�+�n�~�3`��կ-��+�|�bz�����?��vF9���9��/���q�/����n�H`�M&�.Mcu�-�����5Im���k��ǚ�#k7��\��_���f����������=,O����I����K���M�usPF�⣳7?��_�ۏ�6h�s�� %[�e�=�����T������#���L��v���u?c}�k'�y'���o��_[�pj�Q��dǏ�.o�� �5�&Ms�$0�|K=���j�y,��bC����&�e�����Y�ѷ^d����M�J&��1�ƥ9 �yq��?�r��K�K��?�e������o!(:Q9%��J<�Ut�Kɗ)ԥ��m㺂BB�FdHe��/�_�� \��T��\^�Y0��)�a�N��鯮k,�wL���P Mv�[���L����.��L�cTa��Ig���ɼ�<6ѫ������Q8�C;*��i;_ ��^��!p�$N�?�&�����_�����#.���A t %��f�g��>C3`ӹ�j���2����3�z���~UZF�I��������ΧI3m����ng?��k����� s�-B��F������d�U�K��)_ �/��N�Y%�$�Y��'�W��]�{pk-ԏ,�t��X�'6�O=-ʑ�`�9|'�UV�͔~ZC ϝ�[y�ӄ\g����^Iz���Ic4HG�p��O��i�H����;�[k[#Ʊ�D$�5V�V[�r�쭏�>�r��#�|�>]@$}_���-�q��R������/��Ϛ|� 7�5h-ER[ہl(wY%c��eɀ]x�d�����z�潍���(o�uY�WG�=k�Y"=2T��/<�ɒ�f��F!!��1K����Y$���v�T�����Z$�eo��p�p�8�6n���2����/�}>��)z���{]��Y]�.���zrѠ���)>�yf��V���W���H�����/1ү&�T��ԭ��rx�x㓡�}��mV��F������8�@@8���k�5���uu��(k�BD<(���2>ʺ����q������~?��'G��hdɋ-x��<<1�)d���:�O��z���e*���,8~����I�'h�<_u�������@���~#��=��%��.\�� �?\���������z��� f�Kxg�9"1iY��u5�Zl��3����f��DF�׫�2�g�"��h������k��cG� +�+0�!��L��lqDyXG���+W�S������H���7Ec��~���(�}�טd��;���~-~:yBB7���� J����/�4=S�\��#��s,�]=@�DGʻ�]3'U���t�<�\�GՔ����/� B��ٻ3��\���" data-name="ttt.jpg" data-path="../content/Galleries/WCIS/20190118WCISCNYDinner/Performance/ttt.jpg" data-encoded="../content/Galleries/WCIS/20190118WCISCNYDinner/Performance/ttt.jpg" data-ext="jpg" data-isfile="1" data-isimage="1" data-filesize="1 MB" data-date="1547835443" data-sort="0">
where the thumbnails should have appeared.

Then I started to find which photo caused this - if there is one.
In the end, there is ONE particular image. If I upload this single image, above funny codes appears. I thought it is the name, but I tried rename other photos to this, no problem. Then I renamed this particular image to ttt.jpg for example, still it break the page.

This jpg looks fine, and among hundreds images exported by Lightroom. It is 2000x1333 pixels, 1102KB.

What on earth happend?!
 
littlelio
Experienced
Topic Author
Posts: 25
Joined: 20 Nov 2018, 21:21

Re: A particular jpg file breaks the gallery page of a folder

18 Jan 2019, 13:37

OK.

I tried a different size when exporting, still broke the page.

Then I tried a different meta data set. Currently I choose "only copyright", which causes problem. If i change it to "copy and contact info", or "all meta", the jpg would not be creating problem!

It is the first time I know the jpg can cause some problem... how weird...
 
User avatar
mjau-mjau
X3 Wizard
Posts: 13997
Joined: 30 Sep 2006, 03:37

Re: A particular jpg file breaks the gallery page of a folder

18 Jan 2019, 22:54

This definitely looks like something stored in your image's EXIF or IPTC meta data. It could be related to encoding, in which case, it could also be related to your server capabilities (PHP version, encoding). I cannot however exclude that the EXIF in this image is simply corrupt, especially since it's only this image. It certainly looks like a mess.

I can diagnose further, If you can send it to me in PM or by email karl(AT)photo(dot)gallery.
 
littlelio
Experienced
Topic Author
Posts: 25
Joined: 20 Nov 2018, 21:21

Re: A particular jpg file breaks the gallery page of a folder

18 Jan 2019, 23:46

Thanks. Just now I got another image breaks the album page (not the admin backend)

If I upload this image into a folder, the back end is fine, but the album/folder page will never load - keep spinning a red circle.

And I fixed it in the similar way: export from light room but with "copyright and contact info". Again, meta data with "copy right only" will break it.

I will PM you the images.

I am very curious what happened. So the EXIF does matter? However, my other images with the same EXIF settings (copy right only) work well, but just these two images? Something else other than the EXIF matters?

For the backend breaker, I inspected the page elements, it seemed right before "row0" in the file manager, a piece of corrupted code appears.
 
User avatar
mjau-mjau
X3 Wizard
Posts: 13997
Joined: 30 Sep 2006, 03:37

Re: A particular jpg file breaks the gallery page of a folder

18 Jan 2019, 23:52

Please send me a link to your website also. I am pretty sure this is related to a missing  PHP string encoding extension.
 
User avatar
mjau-mjau
X3 Wizard
Posts: 13997
Joined: 30 Sep 2006, 03:37

Re: A particular jpg file breaks the gallery page of a folder

19 Jan 2019, 00:00

I received the image, and could reproduce the issue. I will study it later today when I have time and get back to you.
 
littlelio
Experienced
Topic Author
Posts: 25
Joined: 20 Nov 2018, 21:21

Re: A particular jpg file breaks the gallery page of a folder

19 Jan 2019, 03:45

information sent. Thanks! Curiosity is killing me... :grinning:
 
User avatar
mjau-mjau
X3 Wizard
Posts: 13997
Joined: 30 Sep 2006, 03:37

Re: A particular jpg file breaks the gallery page of a folder

19 Jan 2019, 08:37

Well, basically the "bad" images are containing long string of unidentified IPTC Meta data from some application or other in your workflow. I made a script that outputs the files IPTC data. I was trying to also do some encoding so that the output becomes readable, and perhaps we could guess where it came from, but I don't know the encoding and I'm pretty sure it's something in Chinese anyway.

D85_4152-bad.jpg
Some long string stored in IPTC 25 ["2#025"]
Image

D85_4269-good.jpg
Nothing wrong with any of the "good" files, although it's storing some unknown meta data.
Image

D85_4269-bad.jpg
Next bad file, a lot of junk again, this time stored inside IPTC 219 ["2#219"]:
Image

D85_4152-good.jpg is nice and clean also.

What's going on here? You definitely have some application in your workflow somewhere, that is appending a lot of "junk" to some of the image's IPTC meta data. Or perhaps not garbage (I don't know where it comes from), but at least it's something that really shouldn't be there. Could one of the images have stored some "news" information inside them from some publisher? Those tags 025 (unless that is some news format) and 219 are not meant to be valid IPTC fields:
https://sno.phy.queensu.ca/~phil/exifto ... /IPTC.html

To make things worse, X3 is already using the IPTC [219] ID to store hidden/visible toggle for the image. That's why you have a problem with one of the images in panel ... The two "bad" images aren't even storing IPTC data into the same fields.

Solution
  • Use some app like imageoptim to compress images prior to upload, and also STRIP the IPTC data. I don't see that you need this IPTC data in your images anyway, as they are just increasing the file size.
  • Optionally, you can just disable Settings > Advanced > Store image settings in image as IPTC [screenshot]. This will simply stop X3 from saving or reading the images IPTC, and will therefore fix the issue. The only downside with this, is that if you MOVE an image to another folder, title and description will not be copied alongside the image, because the data is stored in page.json instead. Not really a problem, unless you are creating titles and descriptions AND moving images around in folders.
  • Figure out exactly why some of your images are like this, while most are fine? Probably not worth it if you are just happy with a solution above.
 
User avatar
mjau-mjau
X3 Wizard
Posts: 13997
Joined: 30 Sep 2006, 03:37

Re: A particular jpg file breaks the gallery page of a folder

19 Jan 2019, 08:44

Just for reference, the PHP code to debug IPTC for images. Simply upload it into the folder with the images you want to debug, and then run it in browser debug.php?file=FILENAME.jpg.

debug.php
Code
<?php

getimagesize($_GET['file'], $file_info);
$iptc = iptcparse($file_info["APP13"]);
var_dump($iptc);
 
littlelio
Experienced
Topic Author
Posts: 25
Joined: 20 Nov 2018, 21:21

Re: A particular jpg file breaks the gallery page of a folder

20 Jan 2019, 11:20

Thanks a lot. So IPTC messed it up. Your solution is great, either I can strip the IPTC off the images, or simply turn of IPTC reading and storing in X3. these IPTC info is useless here.

Actually I never touched them in my workflow... I used multiple cameras, perhaps the default IPTC was set differently. For example, the copy right information.

I read that you have extract the IPTC data, and bad image got tons of funny content. I used EXIF program (Opanda IExif, to be exact) on my PC to check them.
* the bad images shows nothing in IPTC section in the program.
* the good images shows some information, suchas the copyright info I set in the camera, which is "SHINEOFLEO". There is another "coded character set" field shows %G and I have no idea what it is.
* However, the other working images, with the same exporting preset with copyright only (which is same as the bad images'), also shows nothing in the IPTC field.

I will try to take another look. Something must be wrong with my lightroom maybe?

Thanks again, at least some actions can be taken to avoid website break-down.
 
littlelio
Experienced
Topic Author
Posts: 25
Joined: 20 Nov 2018, 21:21

Re: A particular jpg file breaks the gallery page of a folder

20 Jan 2019, 11:43

OK I double-checked. The settings for my D850 copy right is nothing. So if export by copy right only, it should be noing. and I just debugged a working image, which shows
Code
bool(false)
The "good" files I sent you, was exported with copy right and contact info, so 
Code
array(3) { ["1#090"]=> array(1) { [0]=> string(3) "%G" } ["2#000"]=> array(1) { [0]=> string(2) "" } ["2#080"]=> array(1) { [0]=> string(10) "SHINEOFLEO" } }
is the contact info, only.
Obviously, those bad images should have nothing in IPTC, but they contains crazy codes. Either lightroom added them or my camera went crazy. I will investigate more if I have time. I think I will remove all IPTC using imageoptim as you advised. 
By the way, I just found by visiting www.mydomain.com/x3/content/ , it will show the whole folder structure, when I tried to debug IPTC. (Perhaps I need to start a new post?) But is it normal for default x3 settings? seems a bit unnerving... any good advice to hide it? I know there are many ways to do it, but by default it was not hidden, this was a bit surprising...
 
littlelio
Experienced
Topic Author
Posts: 25
Joined: 20 Nov 2018, 21:21

Re: A particular jpg file breaks the gallery page of a folder

20 Jan 2019, 12:05

Sorry my bad... I was following the quick start here and completely forgot .htaccess file. Just set it up and locked the folder listing. Perhaps it is good idea to mention in the quick start section?
 
User avatar
mjau-mjau
X3 Wizard
Posts: 13997
Joined: 30 Sep 2006, 03:37

Re: A particular jpg file breaks the gallery page of a folder

20 Jan 2019, 21:34

I'm not quite sure what the remaining issue is now. Is it solved for the time being? I checked the two images you sent, and they don't cause any problems. One doesn't have IPTC, and the other has a few fields unrelated to X3 ... nothing wrong with that. I'm not gonna say X3 is entirely without fault ... Ultimately, your images should not cause errors, regardless of what IPTC is stored inside them ... The main problem, at least for one of the images, is that there was a shitload of data stored inside IPTC #219, which is used by X3 to decide if an image is visible 1 or hidden 0. When output into the panel (which is expecting 1 or 0), that data simply corrupts the HTML. This could in fact be fixed in our code if we converted the output of your IPTC #219 to UTF-8 encoding, but we can't really start doing this for IPTC ID's, which according to the specs, should be empty.

Just for reference, you could open panel/X3.php line 17, and set 'true' as the last parameter, telling X3 to convert encoding to UTF-8.
Code
self::iptc_data($iptc, '219', 'hidden', true)
However, it seems like a better idea to strip the IPTC from your uploaded images or disable IPTC from panel.
littlelio wrote:Actually I never touched them in my workflow... I used multiple cameras, perhaps the default IPTC was set differently. For example, the copy right information.

I read that you have extract the IPTC data, and bad image got tons of funny content. I used EXIF program (Opanda IExif, to be exact) on my PC to check them.
I would have guessed it would come from an application, not your camera, but I really can't say. Seems like a strange IPTC field to store this data from a camera.
littlelio wrote:* the bad images shows nothing in IPTC section in the program.
Not from here either, but that is because most applications that read IPTC will simply display a range of "known" IPTC fields. The strange data was stored in some ID's that are not standard for output.
littlelio wrote:* the good images shows some information, suchas the copyright info I set in the camera, which is "SHINEOFLEO". There is another "coded character set" field shows %G and I have no idea what it is.
Nothing wrong in images storing some data in IPTC, at least as long as it's not a massive amount if encoded "junk" (so to speak) and/or "known" IPTC fields.
littlelio wrote:* However, the other working images, with the same exporting preset with copyright only (which is same as the bad images'), also shows nothing in the IPTC field.
Only the PHP script will see ALL your IPTC fields. Clearly, the bad images had some IPTC fields that the good images did not have. I don't know why, but must be inherited from somewhere.
littlelio wrote:I will try to take another look. Something must be wrong with my lightroom maybe?
I would find that very strange, unless it's from a Lightroom PLUGIN. We have tons of users using Lightroom, and I have never seen this before, and I can't see Lightroom writing that amount of unidentified data into odd IPTC fields.
 
littlelio
Experienced
Topic Author
Posts: 25
Joined: 20 Nov 2018, 21:21

Re: A particular jpg file breaks the gallery page of a folder

20 Jan 2019, 22:02

Hi Karl,

After some time of researching, *I think* I finally findout the root cause.

In short, several factors together got this corruption:
  1. X3 makes use of IPTC information - so you recommended solution of kill the IPTC info would fix it, of course.
  2. NOT the JPG file. I certainly did not add any different information, and I used exiftool and checked, the jpg files are fine - or, the IPTC will be spit loads of crap. However, the to bad jpg files are special - in a ordinary way, but still we have problem because...
  3. the function in PHP, iptcparse is buggy.
I started from the corrupted information. It is good that we have two of them:

4152
Code
array(1) { ["2#025"]=> array(1) { [0]=> string(15721) "=2�Ks���B��m�ŌJ6�2R���J��ʱځ|�a�K��IL"��#/8
4269
Code
array(1) { ["2#219"]=> array(1) { [0]=> string(10955) "z"P�G }���)����d����9z1�W�������ij��w1�5�a�pY�
As you have found from https://sno.phy.queensu.ca/~phil/exifto ... /IPTC.html this IPTC tag is 025(for keywords) and 219(invalid). So I opened both images in a HEX viewer, and I was able to locate the messed up codes (right after {string(xxxx) "}, as below:

4152 first, we pasted a bit more then needed just for reference
Code
94 9B 65 BD 1C 02 19 3D 69 3D 32 CA 4B 73 A2 8F 89 42 AC 9C 7F 6D B1 C5 8C 4A 36 B9 32 10 52 F8 B5 DD 4A CA F0 CA B1 DA 81 7C 04 B7 1C 61 E0 4B B4 08 C2 49 1C 1E 4C 
Note that in UTF-8 code, "=2" hex is "3D 32". If I paste the hex code from 3D 32 in the hex viewer,it is something like:
Code
=2ÊKs¢‰B¬œm±ÅŒJ6¹2RøµÝJÊðʱځ|·aàK´ÂIL"âÿ
 It is quite similar to the spitted mess, although some are funny, even better, if we get the bytes before the 0x3D 0x32, it is
Code
1C 02 19 3D 69
0x3d 0x69 is hex for INT "15721", which is so-called string length in the mess, AND 0x19 is 25 in INT, which is the IPTC tag 025!

This is the pattern, so if we check the other bad jpg:

4269
Code
55 B4 F2 E4 1C 02 DB 2A CB 7A 22 1C 50 C8 47 C2 A0 7D EF FE CB 29 D7 E6 B3 C2 1C 8F 64 F4 06 A5 A8 97 39 7A 31 FF 00 57 F8 E5 FE F5 83 AD EE 99 69 6A 96 F7 77 31 C5
"z"" hex is 7A 22, if we look at the pure mess:
Code
z"PÈG }ïþË)×æ³Âdô¥¨—9z1ÿ
roughly match, but more importantly, the bytes before 7A 22
Code
1C 02 DB 2A CB
again, 0x2A 0xCB is int "10955", and 0xDB is 219! the non-existed IPTC tag!

The reason I am sure the jpgs are innocent, because both good and bad versions of the same image, they have the same codes segment i.e.
1C 02 DB 2A CB xxxxxx for 4269 and 1C 02 19 3D 69xxxxxx for 4152.

Just as I suspected, some part of the raw data in jpg was spitted out, as messed code. The iptcparse made mistakes. but how?

Both starting of the bad code starting with 1C 02. This is important. https://www.codeproject.com/kb/graphics ... 2301&fr=26 explains quite well:
Code
The 8BIM header with the text is divided by even more headers, prefixed by 0x1C 0x02. These blocks then finally contain the information. Multiple blocks with the same type (e.g. Keywords) form a list.

0x1C 0x02 segmentDescription0x1C 0x02 Segment marker (2 bytes)Segment type (1 byte)Segment size (2 bytes excl. marker, type and size)Segment data
The IPTC tag starts from 0x1C 0x02, and I checked the source code of php from https://github.com/php/php-src/blob/mas ... ard/iptc.c (hopefully it is the latest version)
Code
while (inx < str_len) { /* find 1st tag */
 if ((buffer[inx] == 0x1c) && ((buffer[inx+1] == 0x01) || (buffer[inx+1] == 0x02))){
break;
} else {
inx++;
}
}
Yes, it is use 0x1c and 0x02 to find the IPTC tag i.e. the keyword 025 for example.

This is fine, but here is the important part - if read previous paragraph:
Code
Photoshop 6 introduced a slight variation in this header segment. Basically the 4 byte padding has been replaced by a header description text of variable length. The updated sample can now handle these files as well.

8BIM segmentDescription8BIM Segment marker (4 bytes)Segment type (2 bytes)Zero byte padding (4 bytes)Segment size (2 bytes excl. marker, type, padding and size)Segment data
In the lightroom export jpg, the 8BIM segment is easy to find since we just simply search for 8BIM through the hex raw data. The 8BIM segment is defined by photoshop, after a long time search, the page https://www.adobe.com/devnet-apps/photo ... 7409_38034 list all the segment types.
Code
0x04041028IPTC-NAA record. Contains the File Info... information. See the documentation in the IPTC folder of the Documentation folder.
in short, 8BIM with x0404 type is IPTC section, and divided further, x1c02 define the IPTC tags.

If I look at my 4152-good jpg, we have:
  • starting - Photoshop 3.0 8BIM type - x03ED - ResolutionInfo structure.
  • x0404 - the IPTC, followed by x1c02, tag 0 which is applicationrecordversion =4, tag 80 which is by-line, the creator name SHINEOFLEO, which was set in my camera.
  • x040c - Thumbnail resource
  • x0425 - Caption digest. 16 bytes: RSA Data Security, MD5 message-digest algorithm
for 4152-bad
  • same starting - Photoshop 3.0 8BIM type - x03ED - ResolutionInfo structure.
  • x040c - Thumbnail resource
As you may have noticed, there is NO x0404 8BIM type in the bad version! and the fact is, I only exported copyright IPTC in bad version, but there is no copyright IPTC anyway since I have not setup any. In good version, I exported copyright, and contact info, the later is the creator, which makes the 0404 IPTC.

In the php source code,
Code
static char psheader[] = "\xFF\xED\0\0Photoshop 3.0\08BIM\x04\x04\0\0\0\0";
This is the only code used to locate the IPTC type in 8BIM, which I think is not quite right, because:
  • PHotoshop 3.0\08BIM is the 8BIM indentifier, yes
  • but the first type may not be IPTC, there may be something else, such as x03ED - ResolutionInfo structure.
  • I am not 100% sure, but I think this error would just lead to find incorrect segment length
  • what if there is no x0404 8BIM type? just like the 4152 bad?
So here is the catch:

the parser still searching for x1c x02 to locate the iptc tag.

And luckily for 4152 bad and 4269 bad, within the (perhaps wrong segment) length range, those two jpgs has two bytes as x1c and x02, so the piece of code extracted them for the length, and calculated the content - which is garbage.

I will setup my camera with copy right information to fill in the IPTC intead of empty, to try again. and this is only applied for jpg exported from lightroom. A straight out of camera jpg, processed with other software does not have adobe photoshop 8BIM section. Is the photoshop the only IPTC source to add? I don't know, I heard that Nikon just published software to add IPTC.

So far, I think I have dug enough... for me, for now, I would try not to leave x0404 8BIM type empty by setting up my camera with a copyright info - it would affect my workflow the lease. But still, I think the php iptcparse is buggy.

I know this is beyond the x3 website, but since it is image/jpg/IPTC related, I hope this would help anyone comes into similar issue.
Last edited by littlelio on 20 Jan 2019, 22:30, edited 5 times in total.
 
littlelio
Experienced
Topic Author
Posts: 25
Joined: 20 Nov 2018, 21:21

Re: A particular jpg file breaks the gallery page of a folder

20 Jan 2019, 22:06

Hi Karl,

while you were replying, I wrote a long reply as well. So know I saw your reply. Thanks!

All the images are fine and nothing funny happy. The only odd point is, these 2 images have 0x1c 0x02 HEX in the area iptcparser looking for ( i think that's the buggy part, if there is NO 0x04 0x04 8BIM type, the parser should stop early), which is not very high chance for loads of image.

x3 is excellent, so are you!