Increasing industry interest in the optimization of inter-GPU communication has motivated this work to explore new ways to enable peer-to-peer access. Specifically, this paper investigates how reconfigurable optical links between GPUs in multi-GPU servers can allow for minimized memory transfer latencies for given machine learning applications. Silicon photonics (SiP) is proposed as the enabling technology for such a reconfigurable architecture due to the potential for scalable and cost-efficient production. We evaluated our architecture using traffic obtained from an NVLink-connected 8 GPU server executing a set of machine learning models including AlexNet, DenseNet, NASNet, ResNet, MobileNet, and VGG16. Our results show up to 24.91% reduction of the total relative transmission latency (RTL) between peers.