Because of the excellent performance and fast speed of deep neural network, U-Net has become the most popular network framework for medical image segmentation. For various specific image segmentation tasks, researchers have proposed a series of U-Net related methods. However, on the one hand, due to the inherent limitations of convolutional neural networks, the variants of U-Net still cannot model long-range information well while maintaining detailed texture information. On the other hand, since medical images are difficult to obtain a large number of high-quality semantic pixel-level annotations, it is difficult to use supervised deep learning networks. To address these issues above, we proposed a modified U-Net structure and a Gaussian mixture model (GMM) based loss function. This modified U-Net can be well applied to brain MR image segmentation, which can not only restore the detailed information well, but also take into account the relatively large-scale local information. The proposed GMM loss can be used for unsupervised training of neural networks. It effectively alleviates the shortcomings of difficult access to medical image annotation data and improves the performance of deep neural networks. In the experiments in this paper, the GMM loss function can also be used as a regular term to assist supervised learning to achieve better results. Experimental results on brain MR images demonstrate the superior performance of the proposed model.