I'm talking about yellow quests, not orange ones.
And it doesn't seem to be limited to just side missions. Twice I've had a rare animal despawn immediately after dropping off the owl (verified with tagging).
I think the despawn criteria is only checking line of sight and not actually distance from the player which is a problem in survival+permadeath mode where recon is far more important than rushing in.