Preheat oven to 350°F (175°C) with a rack in the center position. Melt 1 tablespoon unsalted butter into a 10-inch cast iron skillet over medium heat; sprinkle the brown sugar evenly throughout the pan. Remove from heat.
Arrange the peach slices in the center of the skillet. Scatter the berries around the sides so that the bottom of the skillet is covered with a layer of fruit. Set aside.
Place the remaining tablespoon butter, oil, and granulated sugar into a mixing bowl; beat with a hand-held mixer until creamy. Add the lemon zest, lemon juice, egg, and egg whites and beat until combined.
Whisk together the flour, baking powder, baking soda, and salt in a separate bowl. Add the flour mixture to the wet ingredients in three parts, alternating with two additions of the buttermilk, until everything is just absorbed. The resulting batter should be sticky and dough-like in consistency.
Carefully spoon the batter over the fruit in the skillet. Bake for 30 to 35 minutes until a wooden toothpick inserted in center comes out clean. Run a thin knife along the edge of skillet to loosen the cake and let the cake cool in the pan on a wire rack.
Once the cake has cooled, invert the pan onto a large serving plate. It should come straight out. If it doesn't tap the bottom of the pan with a knife. Serve warm with vanilla ice cream.