I was playing with a friend earlier, I have my game set to extreme, he joined my game. After a short bit playing we were pushing a compound and he mentioned a red dot near me. It became clear that his game difficulty was set to regular and thus playing a completely different game than I was.

The hosts difficulty should be applied to the group. Simple as that.

If I am in a 4 man and I am set to extreme and 3 people join me with regular settings, there is no point to extreme any longer.