Internet memes have become a very popular mode of expression on social media networks today.
Their multi-modal nature, caused by a mixture of text and image, makes them a very challenging
research object for automatic analysis. In this paper, we describe our contribution to the SemEval2020 Memotion Analysis Task. We propose a Multi-Modal Multi-Task learning system, which
incorporates “memebeddings”, viz. joint text and vision features, to learn and optimize for all
three Memotion subtasks simultaneously. The experimental results show that the proposed system
constantly outperforms the competition’s baseline, and the system setup with continual learning
(where tasks are trained sequentially) obtains the best classification F1-scores.