"No." Is a complete sentence. Stop apologizing like something's wrong when it's perfectly right to tell someone "no." May you be blessed with backbone. Be straight with the other person. Don't worry about letting them down, you're boosting them up and giving the greatest gift
of all - empowerment.