Make the Cookies: In a medium bowl, whisk together the flour, cocoa powder, baking powder and salt; set aside.
Using an electric mixer, cream together the butter, sugar and peanut butter on medium-high speed until light and fluffy, about 3 minutes. Add the egg and beat until combined. Add the vanilla, beat until combined, then scrape down the sides of the bowl. Reduce the speed to low and add the flour mixture in 3 additions, alternating with the milk in 2 additions, scraping down the sides of the bowl as needed. Refrigerate the dough until it is firm, about 1 hour.
Preheat oven to 350 degrees F. Adjust oven racks to upper-middle and lower-middle positions. Line two baking sheets with parchment paper. Scoop out 2 teaspoons dough at a time (I used a small cookie scoop), and place on the prepared baking sheets about 2 inches apart.
Bake until the edges appear dry, 8 to 10 minutes, rotating sheets halfway through the baking time. Remove the cookies from the oven and press a marshmallow half into the center of each cookie. Return to the oven for 2 more minutes. Let the cookies cool on the baking sheet for 5 minutes, then transfer to a wire rack to cool completely.
Make the Frosting: In a medium bowl, whisk together the powdered sugar, butter, cocoa powder, water and vanilla extract. Top each cookie with a heaping teaspoon of frosting, pushing it to the edges so the frosting falls down the sides of the cookie. Top with sprinkles and let sit for at least 30 minutes before serving. Cookies can be stored at room temperature in an airtight container for up to 2 days.