Thank you for the quick response! The first thing I tried was to change the Normal Bias to 0, unfortunately that didn't help. It fixed a few other lighting bugs I've come to expect from unity, but not this one.
I tried your suggestion of baking the lighting, which managed to fix my issue.
However, is there another possible solution to avoid baking the lighting?