In a large skillet, melt the butter over medium. Sprinkle the flour over the butter; cook, whisking constantly, until the flour smells toasty and turns lightly golden brown, about 2 minutes.
Gradually add 3 cups of the milk, whisking to break up any clumps. Add the salt and pepper, and bring to a gentle simmer. Cook, whisking frequently, until the gravy is smooth and thickened, about 3 minutes. Add the remaining 1 cup of milk, if needed, until the desired consistency is reached. Season with more salt and pepper. Serve hot.