Inverting a Colour Selection

When you need to select all colour swatches but one, or select only a few swatches here and there, it might be faster to select the only colour you do not need, in order to select and invert the selection.