A note on bimodality in the log-likelihood function for penalized spline mixed models

For a smoothing spline or general penalized spline model, the smoothing parameter can be estimated using residual maximum likelihood (REML) methods by expressing the spline in the form of a mixed model. The possibility of bimodality in the profile log-likelihood function for the smoothing parameter of these penalized spline mixed models is demonstrated. A canonical transformation into independent observations is used to provide efficient evaluation of the log-likelihood function and gives insight into the incompatibilities between the model and data that cause bimodality. This transformation can also be used to assess the influence of different frequency components in the data on the estimated smoothing parameter. It is demonstrated that, where bimodality occurs in the log-likelihood, Bayesian penalized spline models may show poor mixing in MCMC chains and be sensitive to the choice of prior distributions for variance components.