I never understood why whenever an NPC goes into a building in Fallout 4, there's like a 5 second delay before you can go in. This was never the case in FO3 or NV so why now?

No idea. It was an issue in Skyrim as well. My guess(again, just a guess) would be that the scripting to take you into that cell is linked to an individual door opening animation and when an NPC uses it, you have to reset it before using it. Since 4 and Skyrim use pretty much the same engine(I know, they used the same engine for most of their games, but the versions of Skyrim and 4 seem pretty similar) so they share similar issues.

That's just my guess. Maybe someone else has a better answer.

This is something that really annoys me in Fallout 4. It just makes the monumental loading times even more unbearable.

This was always strange to me in FO4 as well, never understood it either it just always seemed so unnecessary to have the npcs block the door. it has led to some really stupid deaths though.