In this work, we investigate domain adaptation of transformer models to two non-normative variants of the Dutch language: Early modern historical Dutch and contemporary social media Dutch. These two variants of non-normative Dutch share several characteristics that set them apart from normative language, including spelling inconsistencies, semantic shifts and out-of-domain vocabulary. To address these complicating characteristics, we investigate how existing Dutch transformer models can be adapted to both non-normative language varieties. Concretely, we explore two core methodologies for domain adaptation: (1) continued full-model pre-training and (2) training specialized adapters that are added to existing models. The merits of the two approaches are evaluated on two downstream tasks for each domain, namely, sentiment analysis and emotion detection for historical Dutch and emotion detection and irony detection for social media Dutch. Our results demonstrate that both methods for domain adaptation increase performance for social media and for historical Dutch, but that historical Dutch benefits more from domain adaptation. We hypothesize that this is due to current Dutch models having been exposed to a decent amount of social media and less historical data during pre-training. In addition, we compare the performance of domain-adapted transformer models to generative decoder-only models, SOTA for many NLP tasks in other domains, as a potential alternative approach. For historical Dutch, both zero-shot and fine-tuned generative models fail to approximate the performance of our of domain-adapted models. In contrast, fine-tuned generative models can outperform even our domain-adapted social media models for emotion detection and irony detection in Dutch. We conclude that task-specific fine-tuning remains essential to attain decent performance with generative models. The two pretraining corpora used for domain-adapting the transformer models and two novel task-specific datasets for early modern Dutch are published on HuggingFace.