I recently had a conversation with someone who told about a situation when he was asked to do a project by a senior member of the organization. The time frame to prepare was unrealistic, so he respectfully declined.
Others in the organization watched carefully to see how this would play out. There weren’t many people that said no to this person.
How it unfolded ended up being quite positive. She respected my colleague for setting boundaries and for being clear about what he could/could not do. The relationship actually improved with the declined invitation.
Think about your limits and where they are being pushed beyond what is reasonable. Is there something that you should be passing on or rethinking? Saying no is not an automatic career-buster if it is done with good reason and respect.