I read up last week on the Unilateralist's Curse, the problem covered in a brilliant philosophy paper by Cambridge's Nick Bostrom (h/t Buck Shlegeris). The Unilateralist's Curse occurs when a member of a group sharing a common altruistic goal takes an action that hurts the goal because that member mistakenly believes the action to be helpful. If members of a group are each appraising the likelihood of an action being helpful and choosing whether to take the action independently, the action is more likely to happen than it should be.
An example is this: five people have discovered a technology with the potential to cause grave harm and are deciding whether to release it or not. Even if four out of the five decide it is too dangerous, all it takes is one person to release the technology, and so the technology is more likely to be released than it should be since people have mistaken judgment, and the more people there are in the group, the more likely one of them chooses to release it.
Bostrom recommends we resolve the problem by agreeing to a principle of conforming to groups in situations like the above. This of course goes against a modern tendency to praise defiance of groups and avoid doing something simply because others are.
Maybe, in fact, humans already are doing what Bostrom advises, but unconsciously. If people conform more than they should in a situation of solitary rational deliberation, we may actually conform to an optimal degree in the unilateralist's curse situation. If that's the case, acting consciously by a "principle of conformity" would not make as much sense as Bostrom advises, because it would push us over the optimal degree of conformity.
The optimal degree of conformity is hard to know. Are we all more rational than we've been led to believe?