The implied volatility surface is the collection of volatilities implied by option contracts for different strike prices and time-to-maturity. We study factor models to capture the dynamics of this three-dimensional implied volatility surface. Three model types are considered to examine desirable features for representing the surface and its dynamics: a general dynamic factor model, restricted factor models designed to capture the key features of the surface along the moneyness and maturity dimensions, and in-between spline-based methods. Key findings are that: (i) the restricted and spline-based models are both rejected against the general dynamic factor model, (ii) the factors driving the surface are highly persistent, and (iii) for the restricted models option Δ is preferred over the more often used strike relative to spot price as measure for moneyness.