Created attachment 1398709 [details]
reproducer

Description of problem:

When postgresql is compiled with the -O3 option on ppc64 using gcc version 8 it quickly runs into out-of-memory issues due to postgresql trying to allocate more memory for its cache even though the cache is not yet fully used.

It seems that when using -O3 on ppc64 postgresql does not even look at the important part of the cache to see if it is indeed full and just reallocates the cache outright.

I have tried reducing the issue to a simple reproducer (attached).

Version-Release number of selected component (if applicable):
gcc-8.0.1-0.14.fc28.ppc64

How reproducible:
Always

Steps to Reproduce:
1. gcc -O3 oom-reproducer.c -o oom-reproducer
2. run oom-reproducer

Actual results:
Doubles the cache size with each call, runs out of memory quickly.

Expected results:
Doubles the cache size only when all of the cache is used.

Additional info:
If postgresql is compiled with gcc version 7 it works correctly.
Also works correctly when compiling with -O2 instead.

The reproducer is able to reproduce the same issue even on different architectures (x86_64) and gcc versions (7.3) when compiled using -O3. This might be because the reproducer is just smallest set of necessary code pulled from postgresql and run inside a simple never-ending loop.
It does not reproduce the issue when compiled with -O2, however.