The clock drawing test (CDT) is an inexpensive tool to screen for dementia. In this study, we examined if a variational autoencoder (VAE) with only two latent variables can capture and encode clock drawing anomalies from a large dataset of unannotated CDTs (n = 13,580) using self-supervised pre-training and use them to classify dementia CDTs (n = 18) from non-dementia CDTs (n = 20). The model was independently validated using a larger cohort consisting of 41 dementia and 50 non-dementia clocks. The classification model built with the parsimonious VAE latent space adequately classified dementia from non-dementia (0.78 area under receiver operating characteristics (AUROC) in the original test dataset and 0.77 AUROC in the secondary validation dataset). The VAE-identified atypical clock features were then reviewed by domain experts and compared with existing literature on clock drawing errors. This study shows that a very small number of latent variables are sufficient to encode important clock drawing anomalies that are predictive of dementia.