I understand your reluctance to not force people into signing contracts for every cake. I don't, in my area people would run if they had to sign a contract for a birthday cake. But what you can do (and I now do), is set a dollar amount where if the cake is over $200.00 they must sign a contract.
People do understand that, expensive custom work requires a contract. It's a mark of your professionalism that you require it.
I feel for you though, dealing with clients remarks can be hard to deal with and work through. Remember that each time something like that happens you learn from it and become a better business person.... that makes you ready for the next challenging customer.
P.S. Your cake turned out GREAT!