No one is suggesting snow white, broderie anglaise with tan sandals and denim jackets. Most definitely not. Wearing such a pale hue when temperatures turn cooler is a simple matter of weight and fabrication. Cashmere and other wools are a given- what could be chicer than an off-white suit, crafted from the luxe textiles. Coats are on equal sartorial footing, as are pressed collars. Ivory coloured denim works only if paired with heavier knits and outerwear, and we might give light shoes a miss- if only to avoid the inevitable dirt, much worse in winter.
Store the filmy and ultra lightweight for spring (or resort), but keep the weightier pieces. Alone, white is spectral and elegant, paired with noir and the look is edgy and cool. All season, every season.