Breast ultrasounds are a standard modality used to find cancerous lesions and provide early care for patients. However, this modality is sensitive to user dependency. The discrepancies between radiologists’ readings for the same image may lead to inadequate patient care and inconsistent outcomes. As breast cancer continues to affect people around the globe, a new approach is needed to detect cancerous lesions. To provide a system for standardized classifications of these images, we examine the use of state-of-the-art convolutional neural network architectures to improve patient care. To provide confidence in our predictions, we then use local interpretable model-agnostic explanations (LIME) to find what image regions are essential to our model.