file2memory: use a define instead of -1 unsigned value

... to use the maximum value for 'size_t' when detecting integer overflow.
Changed the limit to max/4 as already that seems unreasonably large.

Codacy didn't like the previous approach.