Preheat oven to 350˚F. Cream together butter and sugar. Beat in egg, 1 teaspoon lime zest, and vanilla. In another bowl combine flour, baking powder, baking soda, and salt.
Combine key lime juice and milk in cup. Alternate adding milk mixture and flour mixture to butter mixture. Stir until uniform in color. Batter will be thick.
Pour into greased 8"x5" loaf pan and level with back of a spoon. Bake for 40 minutes or until toothpick inserted in center comes out clean.
Optional glaze: Combine glaze ingredients in small bowl. Add more milk if too thick or more sugar if too thin. Drizzle over completely cooled cake and top with remaining lime zest. Allow to set for 30 minutes before slicing.