Shape memory polymers are fascinating materials, which are able to recover large shape changes. They often find applications in harsh chemical environments which deteriorate their mechanical properties and reduce their life time. The degradation of shape memory polymers can be qualitatively and quantitatively characterised considering their long-term functional behaviour in liquids and gaseous environments. Chemical degradation of polymer materials is caused by small-scale diffusion processes of solvents and gases into the molecular network of the co-polymer. Combining large scale molecular dynamics simulations and experiments, we address here the evolution of thermo-mechanical properties of shape memory polymers when they integrate small molecules by diffusion. Our results show that small molecules enhance the dynamics of structural relaxation and thus contribute to a softening of the material. Via a detailed analysis of the plastic activity and the elastic response, we provide evidence that the same effect is also responsible for increased plasticity and the gradual loss of the shape memory effect.