[Botherer] My father said that as well, and I've met other people who recognised it. Google has "about 57,000" hits for the complete saying in quotes (but only "about 13,200" without quotes).
I heard that, pardon - as my parents used to say, which I've just googled to discover it's from "I Didn't Know You Cared", which I didn't know was written by Peter Tinniswood. [Raak] I don't think so, TBH.