My store has the regular color / size combinations and this behavior is horrible. I have this issue at version 1.7.6.7 and is causing a lot of incorrect orders since I moved from 1.6, so I went ahead and debug the code. After a while I think I'd found the bug and post a PR here: https://github.com/PrestaShop/PrestaShop/pull/20565
It is really late by now but it works. With that you only see what you can buy and it let you switch to other combinations that have stock. It also allows the "allow to order without stock" if you enable it. That part of the code was really complex to find/debug and I'm exhausted, so probably the PR will get rejected by code standards or something like that.
If anyone is willing to test that code and post the comments in the PR or just modify it to apply the PR standards I'll really appreciate the help.