Grind black pepper and fennel seed together in a mortar with a pestle. Pour half the pepper mixture into a small bowl and stir in salt.

Stir remaining pepper mixture, garlic, rosemary, 2 tablespoons olive oil, orange juice, and cayenne pepper together in a bowl.

Rub rosemary mixture over both sides of flank steak, poking mixture into steak several times with a fork. Transfer steak to a resealable plastic bag and marinate in the refrigerator for 2 hours. Remove steak from marinade; scrape off and discard any excess marinade.

Preheat an outdoor grill for medium-high heat and lightly oil the grate.

Sprinkle reserved pepper-fennel mixture over steak. Drizzle 1 tablespoon olive oil over orange halves.

Cook steak on the preheated grill until it begins to firm and is reddish-pink and juicy in the center, about 5 minutes per side. An instant-read thermometer inserted into the center should read 130 degrees F (54 degrees C). Remove to a plate to rest for 10 to 15 minutes before slicing across the grain.

Place oranges, flesh-side down, on the grill and cook until flesh is golden and caramelized, 3 to 5 minutes. Place steak and 3 orange halves on a serving platter. Squeeze remaining caramelized orange half over the steak.