It doesn´t distort more on the individual distortion modules but voice+voice+voice...etc... adds up in volume and distorts the audio out which is by default set to hard clipping and adds tons of distortion (mostly in an unpleasant way) if driven too hard... lowering the input of the individual voices going into the audio out module should help...
Second (didn´t tested... might be nonsense): if you have multiple voice basically playing the same (or very close) distorted thing the distortion artifacts get more pronounced which can give the feeling of more distortion even if the individual voices sounding fine...