Detecting anomalies accurately in time series data has been receiving considerable attention due to its enormous potential for a wide array of applications. Numerous unsupervised anomaly detection methods for time series have been developed because of the difficulty of obtaining accurate labels. However, most existing unsupervised approaches suffer from the problem of anomaly contamination, which results in models that are unable to learn the normal pattern well and further deteriorate the performance of detection methods. To this end, a novel unsupervised method, called Self-adversarial Variational Autoencoder with Spectral Residual (SaVAE-SR), is introduced for time series anomaly detection in this paper. The SaVAE-SR first produces labels for unlabeled training data using the spectral residual technique to identify the most critical anomalies. A VAE model with a modified loss that can leverage label information to remove the influence of anomalous points is then trained in a self-adversarial manner, enabling the model to self-evaluate the learning of complex data distribution and improve itself accordingly. Specifically, the encoder acts as an encoder to approximate the posterior of latent variables and as a discriminator to evaluate the generative ability of the generator and improve itself accordingly. The generator is trained to capture the underlying data distribution and attempts to produce real samples to deceive the discriminator. The encoder and generator of the model compete with each other just like the behavior of GANs but work together under the theoretical framework of VAEs. As a result, the SaVAE-SR model combines the respective strengths of the VAE and adversarial training but does not require an additional discriminator, which makes the whole model very compact. Extensive experiments on five datasets demonstrate the superiority of the proposed method over the existing state-of-the-art methods.