Fitness in biology does not refer to how strong or large or intimidating or predatory a certain species is, it refers to how a species form is advantageous to their environment and more importantly how advantageous it is if that environment changes, so large size is not necessarily a good thing for when the environment changes, see what happened to the dinosaurs.