Generaly speaking it does matter.
The top colour should be on the wearer's right (our left as viewed) and the bottom colour should be on the wearer's left.
It should be noted, however, what happens on the back?
Strict rules of heraldry (as with flags and banners) dictate the the colours should reverse. i.e. the colours go over the wearer's shoulders and continue down his back.
However this gets complicated with, say, Beaufort (white blue) and Richard of York (blue white), as viewed from behind the wearer is now playing for the other team. This MIGHT explain why livery badges worn on the back tended to be bigger! It stopped you getting cut down by your own team from behind.
This may also explain why 'bends' (coloured bands worn diagonally) were worn. Edward IV issued red bends on 1461. It may also explain why Edward, George and young Richard switched to blue/claret (wine red) around the same time. Partly this removed the chances of confusion with Beaufort while changing the white side to the colour of dried blood emphasised the deaths of their father Richard of York and their brother Edmund at Wakefield in 1460. The new livery colour advertised their loss.
There is also the suggestion that Richard of York was lured out of Wakefield Castle (and to his death) due to false colours being displayed. If you get hold of a Beaufort white/blue livery jacket and wear it back-to-front it becomes a Yorkist blue/white!
I would add there are several suggestions about why Richard came down from Wakefield castle and this is only one of them.
My pictures of Sandal/Wakefield Castle are here: link
Barry