Search…

X3 Photo Gallery Support Forums

Search…
 
rampageX
Posts: 7
Joined: 25 Nov 2021, 00:32

Re: Sorting is inconsistent in some locales when files have mixed character-sets

03 Feb 2022, 22:28

mjau-mjau wrote: * continued from my last post

5. PHP sort
I notice that strnatcasecmp() sorting from PHP might offer better sorting in some cases. It's difficult to use PHP for "live" sorting from the interface (without re-loading from server), but I could perhaps store the PHP sort on initial folder load and use that as reference when sorting "live". I don't know how well this will work, but I might include as an option.
I think this is a good idea.
 
User avatar
mjau-mjau
X3 Wizard
Posts: 13993
Joined: 30 Sep 2006, 03:37

Re: Sorting is inconsistent in some locales when files have mixed character-sets

03 Feb 2022, 23:09

rampageX wrote:Pure chinese words sorting is difficult, the 'perfect' way is:
1. Sort chinese words by 'PINYIN' first, as '非洲'=FeiZhou, '中国'=ZhongGuo,  F < Z, then 非洲 will be front of 中国;
2. if chinese wors have same 'PINGYIN', as '中国'=ZhongGuo, tha same as '中过'=ZhongGuo, we must sort these two words with Unicode Table.
Thanks. We kinda need a programmatic way to do sorting from Javascript, preferably not slow. I am guessing that localeCompare() (used in current Files app version) works with Chinese as long as there is ONLY Chinese. Or maybe you need to specifiy 'zh-CN' (etc) as locale option. This is after all exactly what localeCompare() function in Javascript was made for.
rampageX wrote:I was tried TinyFileManager, the Chinese words sort was wrong too, but there were always stay behind all the latins, and all latins sort keep correct.
That's because it uses basic sort method as in previous Files version, which I will include as an OPTION in next release:
mjau-mjau wrote:3. New option basic sort
This is the sorting method used in previous Files app, and was reported working fine with Chinese by both @amwpsaa and @rampageX. Basically it just involves lowercase a < b sorting, which resolves nicely in modern browsers and is fast. What this sort method does NOT do: 1. It does not understand that 2<10 (it just looks at first letter), and 2. It cannot sort mixed unicode (it will sort ["a", "z", "é"] instead of ["a", "é", "z"]. In many cases however, this will work nice and fast for many users who are not picky about these two flaws.
As you can see in your own screenshot in array #3, latin is sorted correctly and comes before Chinese. It fails to sort "extended" latin (éüø) correctly, and does not understand numbered names 2<10. I don't want to use/build a slow custom JS sorting function for this, since localeCompare() was made exactly for this, and a custom function wouldn't solve Chinese anyway.

Thanks for all the feedback!
 
User avatar
mjau-mjau
X3 Wizard
Posts: 13993
Joined: 30 Sep 2006, 03:37

Re: Sorting is inconsistent in some locales when files have mixed character-sets

05 Feb 2022, 07:30

New version 0.4.1
Please check sorting. I think "inconsistent sorting" / "sorting chaos" is resolved.
https://demo.files.gallery/?tests/sorting

Let me know!
 
rampageX
Posts: 7
Joined: 25 Nov 2021, 00:32

Re: Sorting is inconsistent in some locales when files have mixed character-sets

05 Feb 2022, 08:09

mjau-mjau wrote: New version 0.4.1
Please check sorting. I think "inconsistent sorting" / "sorting chaos" is resolved.
https://demo.files.gallery/?tests/sorting

Let me know!
Image
 
User avatar
mjau-mjau
X3 Wizard
Posts: 13993
Joined: 30 Sep 2006, 03:37

Re: Sorting is inconsistent in some locales when files have mixed character-sets

05 Feb 2022, 09:59

rampageX wrote:Image
In your screenshot on the RIGHT, I see that all latin characters+unicode are sorted correctly, and it's now consistent (when navigating and re-sorting), and that is the fix. On your system (Chinese OS browser, I don't know what combo), it is sorting Chinese BEFORE Latin, because browser/device/javascript thinks that is correct based on your own locale (Chinese). This is a device implementation issue. On my system, it sorts Latin FIRST, because my system is detected Latin. It's your OS/browser/Javascript that thinks the Chinese should be on top.
Image

There is no Javascript method that sorts like you request, unless you can configure it by specifying a locale intl.Collator().

In your screenshot on LEFT, the dirs are loaded from PHP and don't have or need sorting by Javascript. The reason they seem correct (for you), is just because PHP does not recognize unicode (Chinese) and therefore just puts them last. This sorting is actually WRONG because "écru" is after "zzz". We cannot use PHP sorting for dynamic javascript list sorting, especially when it's not even correct.

There is a new option sort_function in latest release. By default, it is set to "locale", but you can change it to "basic" and it will work like datatables.js and like Files app before version 0.3.0 (you tested and said it worked). It will sort basic latin correctly, but will not understand é<f (unicode) or 2<10, and Chinese words will come last, but only because all unicode is places last.

In conclusion, the "Sort chaos" you are referring to no longer exists, and sorting is working as it's supposed to based on device/browser implementation. This might not be right for Chinese on your OS/browser, but then you have new sort_function option. There is no way to sort Chinese in Javascript like you request. If you revert to sort_function "basic", it will sort like in the menu ... The only reason this works, is because it doesn't recognize unicode properly and will place unicode words last.
 
rampageX
Posts: 7
Joined: 25 Nov 2021, 00:32

Re: Sorting is inconsistent in some locales when files have mixed character-sets

05 Feb 2022, 18:44

mjau-mjau wrote:
rampageX wrote:Image
In your screenshot on the RIGHT, I see that all latin characters+unicode are sorted correctly, and it's now consistent (when navigating and re-sorting), and that is the fix. On your system (Chinese OS browser, I don't know what combo), it is sorting Chinese BEFORE Latin, because browser/device/javascript thinks that is correct based on your own locale (Chinese). This is a device implementation issue. On my system, it sorts Latin FIRST, because my system is detected Latin. It's your OS/browser/Javascript that thinks the Chinese should be on top.
Image

There is no Javascript method that sorts like you request, unless you can configure it by specifying a locale intl.Collator().

In your screenshot on LEFT, the dirs are loaded from PHP and don't have or need sorting by Javascript. The reason they seem correct (for you), is just because PHP does not recognize unicode (Chinese) and therefore just puts them last. This sorting is actually WRONG because "écru" is after "zzz". We cannot use PHP sorting for dynamic javascript list sorting, especially when it's not even correct.

There is a new option sort_function in latest release. By default, it is set to "locale", but you can change it to "basic" and it will work like datatables.js and like Files app before version 0.3.0 (you tested and said it worked). It will sort basic latin correctly, but will not understand é<f (unicode) or 2<10, and Chinese words will come last, but only because all unicode is places last.

In conclusion, the "Sort chaos" you are referring to no longer exists, and sorting is working as it's supposed to based on device/browser implementation. This might not be right for Chinese on your OS/browser, but then you have new sort_function option. There is no way to sort Chinese in Javascript like you request. If you revert to sort_function "basic", it will sort like in the menu ... The only reason this works, is because it doesn't recognize unicode properly and will place unicode words last.
I got it, and understand, thank you for your hard work.
 
User avatar
mjau-mjau
X3 Wizard
Posts: 13993
Joined: 30 Sep 2006, 03:37

Re: Sorting is inconsistent in some locales when files have mixed character-sets

12 Feb 2022, 23:40

Just to conclude: New version 0.4.0 resolves sort chaos / sorting inconsistency, and new sort_function allows you to choose between "locale" or "basic" sort, or set a locale string for the localeComparison() function. I don't know about mixed Chinese+latin, but latin, unicode-latin and numbered names are definitely sorted correct and consistently. To sort Chinese correctly, you might try 'basic' sort_function or try to pass a China-specific locale string.