Preheat the oven to 425 degrees F (220 degrees C). Grease a casserole dish.
Wash cod and pat dry with paper towels. Place 2 rosemary sprigs inside the belly of the cod. Season inside and outside of cod with salt and pepper; place in the prepared casserole dish.
Bring a pot of water to a boil. Cut an 'X' into the bottom of 4 of the tomatoes and submerge in boiling water for 20 seconds. Remove from the hot water and peel off skin. Cut in half horizontally and place tomato halves around the cod. Dice remaining tomato and scatter between the cod and the tomato halves. Season with salt and pepper.
Sprinkle remaining rosemary, black olives, green olives, garlic, and fennel seeds over cod and tomatoes. Drizzle olive oil on top.
Bake in the middle of the preheated oven until cod flakes easily with a fork, about 25 minutes. Drizzle balsamic vinegar over tomatoes before serving.