The most common phrase in spoken English to show that the speaker recognises that his/her reaction is in some way unhelpful or unwelcome is "I'm afraid" or "I'm afraid no". It may warn of disagreement, but its general meaning is wider and indicates the speakter sees his/her reaction as unavoidably unhelpful.
Afraid not
The most common phrase in spoken English to show that the speaker recognises that his/her reaction is in some way unhelpful or unwelcome is "I'm afraid" or "I'm afraid no". It may warn of disagreement, but its general meaning is wider and indicates the speakter sees his/her reaction as unavoidably unhelpful.