No, they would not notice at all. My red velvet is a favorite with many of my customer and I have made it with fat free and regular on several occassion and I can't even tell the difference, neither did they.
okay, so i just went to the store to get regular fat buttermilk and what do you know? they don't carry it anymore. the guys said that they didn't carry it with the major/regular milk brands in our area.