The International Agency for Research on Cancer announced that in 2020 female breast cancer became the most diagnosed cancer worldwide and the most common cause of cancer-related death in women. Still, breast cancer generally has a good prognosis with timely detection and appropriate treatment. Recently, computer-aided diagnosis (CAD) systems have shown promising results in using artificial intelligence (AI) to detect malignant lesions in breast ultrasound (US) imaging. When working with AI in a clinical setting, however, the American College of Radiology advocates for radiologist understanding of the algorithms in use. Accordingly, this study contributes to an ongoing collaboration between the University of Wisconsin-La Crosse and Mayo Clinic Enterprise (BUS Project) by investigating three methods of AI explainability for the CAD software in development. Class activation maps, saliency maps, and attention map-enhanced class activation maps are compared to determine the most useful technique for visualizing regions in the US used to determine pathology. These visualizations could provide a radiologist with confidence in the black box algorithm when the lesion is localized, and reveal limitations in the model prediction when extraneous regions are highlighted. As more data becomes available and the CAD system is improved, it is the hope of this study that the BUS Project continues to develop and prioritize model explainability for the sake of responsible AI in healthcare.