I derive the finite-sample bias of the conditional Gaussian maximum likelihood estimator in ARMA models when the error follows some unknown non-normal distribution. The general procedure relies on writing down the score function and its higher order derivative matrices in terms of quadratic forms in the non-normal error vector with the help of matrix calculus. Evaluation of the bias can then be straightforwardly conducted. I give further simplified bias results for some special cases and compare with the existing results in the literature. Simulations are provided to confirm my simplified bias results.