There is no such option as "gallery layout" or "folders layout". There is the "gallery module", which controls layout of the images on a page, and there is the "folders module", which controls the layout of folders on a page.
You don't "choose" on or the other, and none of them "wins" over the other. If you have IMAGES in a folder, they will display in the page with settings from the GALLERY LAYOUT. If you have FOLDERS in a page, they will display with settings the FOLDERS layout.
Here is a page with child FOLDERS, therefore settings from
FOLDER module are displaying:
https://demo.photo.gallery/galleries/
Here is a page with IMAGES, therefore settings from
GALLERY module are displaying:
https://demo.photo.gallery/galleries/nature/
If you have BOTH images AND subfolders on the same page (not normal), then the page will display both FOLDERS module and GALLERY module.