In a small bowl, beat the cream cheese with an electric mixer on low speed until smooth. If you're having trouble getting the cream cheese to smooth out, leave it at room temperature for a few minutes to soften.
Gradually add the sour cream or yogurt, beating the mixture until fully combined.
Add the brown sugar, and vanilla; beat just until combined.
Stir in enough milk to make dipping consistency. Add it a little at a time to avoid thinning out the dip too much.
Cover and chill for at least 1 hour before serving. This dip can be made up to 24 hours before serving. Serve the fruit dip with cream cheese alongside assorted fruit, berries, cookies, or graham crackers.
