Combine flour, baking powder, baking soda, nutmeg and salt in a medium bowl. Stir with a fork to mix well and set aside. Combine milk and vanilla in small bowl and set aside.
In a large bowl, beat the butter,sugar and brown sugar together at high speed until light and well combined, stopping once or twice to scrape down the bowl. Add the eggs, one at a time, beating well each time. Add the mashed sweet potatoes and mix at low speed for 1 minute, or until the batter is evenly mixed.
Add about half of the flour mixture and beat gently at low speed, until flour disappears into batter. Add half of the milk mixture and mix gently to combine everything well. Mix in the remaining flour, then the remaining milk, beating gently only until you have a thick, smooth batter.
Scrape the batter into the prepared tube pan and bake at 350 for between 60 and 75 minutes or until the cake is evenly browned, springs back when touched gently in the center, and a wooden skewer inserted in the center comes out clean.
Cool in the pan on wire rack or a folded kitchen towel for 20 minutes. Turn out cake onto wire rack, top side up, and cool to room temperature. Enjoy!