I don't think anyone has to worry about being "the only one" not wearing gold. There will be thousands who aren't wearing it. If you don't have anything and don't want to spend $ on it, you won't feel out of place at all. It's different when they hand out shirts to everyone at the game (like the "white outs" at basketball games), then you might stand, out but this is different.