This paper describes the system we developed for the shared task “Hero, Villain and Victim: Dissecting harmful memes for Semantic role labeling of entities” organized in the frame- work of the Second Workshop on Combating Online Hostile Posts in Regional Languages during Emergency Situation (Constraint 2022). We present an ensemble approach combining transformer-based models and linguistic information, such as the presence of irony and implicit sentiment associated to the target named entities. The ensemble system obtains promising classification scores, with a macro F-score of 55%, resulting in a third place finish in the competition.