Preheat the oven to 375 degrees. Spray the cast iron skillet with oil and then lightly dust with flour to prevent the cake from sticking.
In a mixing bowl, add 2 1/2 cups of flour, 2 tsp of baking powder, 1/2 tsp baking soda, 1/2 tsp salt, 1 tsp of ginger, and 1/2 cup of sugar. Whisk together until all the dry ingredients are combined.
In a separate mixing bowl, whisk together 2 eggs, 1 1/2 cups of buttermilk, 1/4 cup melted butter, 1 tsp vanilla, and 1/2 tsp almond extract. Pour the wet ingredients into the dry ingredients and mix thoroughly until no dry ingredients remain.
Pour the cake batter into the prepared cast iron skillet and smooth out evenly. Place the sliced figs all over the top of the batter, pressing them lightly down in. Next, pour the almond slivers all over the top of the cake. Finish by evenly sprinkling three packets (about 3 tsp) of Sugar in the Raw over the top of the cake
Place the skillet on the middle rack of the oven and bake at 375 degrees for 45-50 mins. The cake is done when it's golden brown and the middle is set. If you stick a toothpick in the middle it should come back out clean.