In messing about with the vertex toolbar code, I stumbled across how colors are set for the swatch previews and noticed that they inaccurately display what color they are set to. To fix this, and the performance seems alright up until about 30 swatches, is to generate 1 or 2px textures and set their colors instead.
You would comment out any GUI.color code inside the loop drawing the swatches, and add this:
Texture2D swatchTex = new Texture2D( 2, 2, TextureFormat.ARGB32, false, false );
for( int x = 0; x < swatchTex.width; x++ )
for( int y = 0; y < swatchTex.height; y++ )
swatchTex.SetPixel( x, y, USER_COLORS[i] );
swatchTex.Apply( false, false );
and then draw it normally with the existing DrawPreviewTexture() call.
EditorGUI.DrawPreviewTexture( layoutRect, swatchTex, null, ScaleMode.StretchToFill, 0 );
This will then display accurately what color the user has set without noticeable overhead.