In theory if these “modifying” tokens have value as interactions they should be learned by the model already.