According to all my calculations, fluoride burns hotter than oxygen. Let's consider the simple case of combustion with hydrogen. To make things easy, we'll use gaseous reactants at 298.15 K, a stoichiometric mixture, and no dissociation.
First, oxygen and hydrogen:
O2 + 2 H2 --> 2 H2O
The enthalpy of the reactants must equal the enthalpy of the products. The standard enthalpy of formation, i.e. Hfo at 298.15 K, of gaseous molecular oxygen and hydrogen is zero. Therefore, the enthalpy on the right side of the equation is zero, thus that on the left side must also be zero.
At 298.15 K the enthalpy of water vapor is –241.83 kJ/mol. As water heats up its enthalpy increases. To make the equation balance, the temperature of the water on the right side of the equation must be that which will make its enthalpy equal to that of the reactants, i.e. zero. Water's enthalpy is equal to zero at a temperature of about 5,000 K.
Now fluoride and hydrogen:
F2 + H2 --> 2 HF
Like oxygen and hydrogen, the standard enthalpy of formation of gaseous molecular fluoride is zero; therefore, as before, the enthalpy of the products must be zero. At 298.15 K the enthalpy of HF is –272.55 kJ/mol. The standard tables and equations for calculating enthalpy generally go to only 6,000 K, but by extrapolating the curve, I estimate the enthalpy of HF is zero at about 7,800 K.
In practice, the temperatures would be lower than this due to dissociation. Furthermore, rockets don't use a stoichiometric mixture, thus further lowering the temperature. Nonetheless, the above demonstrates that the combustion of F with H will produce higher temperatures than the combustion of O with H.
The exhaust gas velocity of a rocket engine, and thus the specific impulse, is roughly proportional to (T/M)1/2. For the above two reactions we get:
(5000/18)1/2 = 16.67 and (7800/20)1/2 = 19.75
Remember, however, this is an over-simplified scenario. When taking into consideration dissociation and realistic mixture ratios, fluoride will out perform oxygen by about 5% when burned with hydrogen. When burned with hydrazine, fluoride does about 12% better than oxygen.
(edit) Something else that further lowers the flame temperature is that the propellants in a rocket are cryogenic liquids rather than gases. Some of the heat of combustion has to vaporize the propellants and raise their temperatures, thus resulting in a lower final temperature.