Preheat oven to 400°. Toss lemon slices and 2 Tbsp. oil on a rimmed baking sheet; roast until caramelized and lightly charred in a few spots, 10–12 minutes; let cool.
Combine lemon slices, feta, chiles, bay leaves, lemon juice, and remaining ½ cup oil in a small dish; season with pepper. Cover; chill 12 hours.
Do Ahead: Feta can be made 5 days ahead. Cover and chill.
