I think I'm already aware of this "limitation", and looks like I'm going to have to fix it somehow. It works fine if using the CREATION DATE of the folders, but doesn't work properly with the custom page DATE input setting. You might have wait until next minor release for this to be fixed. In the meantime, you may need to disable pagenav or add a class that hides pagenav for your blog pages.
Reason? Page navigation is a bit tricky. For all the other SORT options (of a parent folder), we simply compare names, titles or creation dates of all sibling folders to create the correct sort output. However, for custom dates (as in your case), for EACH of your posts, we would need to additionally load the page.json data file for ALL sibling pages, to extract the custom date setting and sort everything appropriately. This may slow down processing ... but then again, it won't affect speed of cached pages. I will have it fixed!