Red Teaming Multimodal Language Models: Evaluating Harm Across Prompt Modalities and Models