Christine Carter has a blog post, 21 Ways to "Give Good No", at Greater Good in which she deals with this issue. I wasn't so interested in the "21Ways" that appear in Part Two of the post. Parts One and Three were another thing.
Plan How Much Time You Can Take Away From Work
In Part One of her article, Carter says it's easier to say no "when we have a concrete reason for doing so—a way to justify our refusal." "...we need to create the reason for saying no before we need it—we need a decision making structure, or “rules” to guide us so that we don’t have to agonize over every invitation." She talks about planning ahead for how many social invitations you can accept during the course of the week, then saying no to the rest. Also plan when you're going to work and say no to any requests that will conflict with your work time.
I've written here about writers working for free and being asked to work for free. If you want to be able to support some organizations with free work or appearances, you can plan ahead for how much of your time you can afford to give away. When you've reached your limit, say no to additional requests.
Your Decision Is Made. Move On.
In Part Three, Carter says, essentially, say no and stop thinking about it. "...when we make a decision in a way that allows us to change our minds later, we tend to be a lot less happy with the decisions that we make." Perhaps because the decision isn't really made if we can make a different one down the line? Forget about being happy with a decision. If the door is left open, the decision is still hanging over you, is it not? How time consuming and energy depleting is that?
So make your plan so you can make a decision. And when you've made a decision, get back to work.