I too get the idea of installing server and building from there. While my main PCs run Mint 13 with KDE and Mint 11 Gnome, I do have an old eeepc that I installed Ubuntu Server on and then XFCE and Chromium, because a full Xubuntu install was too large for the 2GB SSD.
i5 4440 3.1Ghz, Asus H87M-E motherboard, 8GB DDR3, 2GB DDR5 nVidia card, 2x500GB Seagate hd, Coolermaster 750W PSU running Mint 16 MATE and Win7
eeepc 1015PX, 1.66 Ghz Intel atom processor, 2 GB DDR3, 320 GB hd, Mint 14 MATE. Fan needs fixing.