One of the major challenges currently facing the field of argumentation mining is the lack of
consensus on how to analyse argumentative user-generated texts such as online comments. The
theoretical motivations underlying the annotation guidelines used to generate labelled corpora
rarely include motivation for the use of a particular theoretical basis. This pilot study reports on
the annotation of a corpus of 100 Dutch user comments made in response to politically-themed
news articles on Facebook. The annotation covers topic and aspect labelling, stance labelling, argumentativeness detection and claim identification. Our IAA study reports substantial agreement
scores for argumentativeness detection (0.76 Fleiss’ kappa) and moderate agreement for claim
labelling (0.45 Fleiss’ kappa). We provide a clear justification of the theories and definitions
underlying the design of our guidelines. Our analysis of the annotations signal the importance of
adjusting our guidelines to include allowances for missing context information and defining the
concept of argumentativeness in connection with stance. Our annotated corpus and associated
guidelines are made publicly available.