Heat 2 tablespoons oil in a large skillet over medium-high heat. Add halibut fillets; season with salt and pepper. Cook until flesh flakes easily with a fork, about 4 minutes per side. Transfer to a plate; cool until easily handled, about 5 minutes. Flake into small chunks.
Whisk mayonnaise, egg, and mustard together in a large bowl. Mix in oregano, thyme, and rosemary. Stir in flaked halibut and bread crumbs. Shape mixture into 6 cakes.
Heat remaining 2 tablespoons oil in the skillet over medium-high heat. Add halibut cakes and cook until browned, about 5 minutes per side.