Place a Dutch oven or large pot over medium heat. Add the chopped bacon and cook, stirring occasionally, until crisp, about 7 to 9 minutes. Use a slotted spoon to transfer the bacon to a paper towel-lined plate, leaving about 2 tablespoons of rendered fat in the pot.
Add the diced onion and sliced celery to the pot. Sauté in the bacon fat until softened, about 8 minutes. Add the minced garlic and cook for 1 more minute until fragrant.
Pour in about 1/2 cup of the chicken broth to deglaze the pot, scraping up any browned bits from the bottom with a spoon.
Add the cauliflower florets, the remaining 3 1/2 cups of chicken broth, salt, pepper, and the bay leaf. Stir to combine.
Bring the soup to a simmer, then reduce the heat to low, cover, and let it cook for 15 to 20 minutes, or until the cauliflower is very tender and easily pierced with a fork.
Remove the pot from the heat and take out the bay leaf. Use an immersion blender to carefully purée the soup directly in the pot until it's smooth and creamy.
If the soup is thicker than you'd like, stir in a little more broth until it reaches your desired consistency. Taste and adjust seasoning with more salt and pepper if needed.
Ladle the soup into bowls. Top with the reserved crispy bacon, shredded cheddar, a dollop of sour cream, and a sprinkle of chives before serving.