Place bacon in a large saucepan and cook over medium-high heat until crispy, about 10 minutes. Remove bacon and crumble into a bowl. Reserve bacon grease in the skillet.
Place onions, celery, butter, thyme, marjoram, nutmeg, and chile powder in the still-hot saucepan with the bacon grease. Cook until onions are soft, 5 to 7 minutes. Add chicken stock and potatoes; bring soup to a boil. Reduce heat to medium-low; cook until potatoes are soft, 20 to 25 minutes.
Puree the soup with an immersion blender. Season with salt and pepper; garnish with crumbled bacon.