Cook bacon in a large Dutch oven or heavy-bottomed pot over medium-high heat, stirring occasionally, until crisp, about 20 minutes. Using a slotted spoon, transfer the bacon to a paper towel-lined plate.
Reduce the heat to medium and add the onion, shallot and garlic to the rendered fat in the pot and cook, stirring occasionally, until the onion is completely softened, about 10 minutes. Return the bacon to the pot, then stir in the coffee, water, vinegar, maple syrup, brown sugar, honey, allspice and chili powder. Bring the mixture to a simmer (lower the heat if you see that it's actually boiling) and cook, stirring occasionally, until the mixture thickens and a rubber spatula or wooden spoon leaves a distinct trail when dragged across the bottom of the pot, 1 to 1½ hours.
Remove from the heat and allow the bacon mixture to cool for 15 minutes. Using a slotted spoon, transfer the mixture to the work bowl of a food processor. Discard any excess fat in the pot. Pulse the bacon mixture until finely chopped, about 5 pulses, or to desired consistency. Store the jam in a glass jar with a tight-fitting lid in the refrigerator for up to 2 weeks. The jam is best served at room temperature.
Food Processor Alternative: You can pulse in a high-powered blender, or chop/smash very finely with a knife to get the correct consistency.
Bacon: THE most important ingredient in this recipe. I suggest using thick-cut bacon to make the jam.
Coffee Alternative: If you can't have caffeine, decaf will work just fine. If you need to avoid coffee entirely, substitute the same amount of water.
Storage: Keep freshly made bacon jam in the refrigerator for up to 2 weeks in a glass container. Serve at room temperature or warm a little before serving.
Freezing: For longer storage, freeze this savory jam. Leave some headspace at the top of the jar when you fill it up and store it in a freezer-safe container for up to 3 months.
Thawing: Transfer to the refrigerator to thaw and keep for up to 2 weeks after moving to the refrigerator.