Preheat oven to 300 degrees F. Place the bread on a wire rack set in a rimmed baking dish. Bake bread until almost dry throughout (center should remain slightly moist), about 16 minutes, flipping slices halfway through cooking. Remove bread from rack and let cool for 5 minutes. Return baking sheet with wire rack to oven and reduce temperature to 200 degrees.
In a medium bowl, whisk together the milk, egg yolks, brown sugar, ½ teaspoon cinnamon, 2 tablespoons melted butter, salt, vanilla extract, Triple Sec, and orange zest until well blended. Stir in the sliced almonds. Transfer the mixture to a 9x13-inch baking dish. In a small bowl, stir together the 4 tablespoons granulated sugar and 2 teaspoons ground cinnamon; set aside.
Soak the bread in the milk mixture until saturated but not falling apart, about 20 seconds per side, ensuring that some almonds adhere to the bread. Using a slotted spatula, pick up a bread slice and allow excess milk mixture to drip off; repeat with remaining slices. Place soaked bread on another baking sheet or platter. Sprinkle 1 to 2 teaspoons of the cinnamon-sugar topping over one side of each slice of soaked bread.
Heat ½ tablespoon unsalted butter in a 12-inch skillet over medium-low heat. When the foaming subsides, use the slotted spatula to transfer 2 slices of the soaked bread to the skillet, cinnamon-sugar side down first, and cook until golden brown, 3 to 4 minutes. While they cook, sprinkle the top of the bread with 1 to 2 teaspoons of the cinnamon-sugar mixture. Flip and continue to cook until the second side is golden brown, 3 to 4 minutes longer. (If the toast is cooking too quickly, reduce the temperature slightly.) Transfer to the baking sheet in the oven. Repeat cooking with remaining bread, 2 pieces at a time, adding ½ tablespoon butter for each batch. Serve warm, sprinkled with powdered sugar, and/or drizzled with maple syrup.