Learn 2D filter banks are currently being used in high-impact applications such convolutional neural networks, convolutional sparse representations, etc. However such filter banks usually have plentiful filters, each being non-separable, accounting for a large portion of the overall computational cost. In this paper we propose a novel and computationally appealing alternating optimization based algorithm to estimate a linear combination of separable (rank-1) filters to approximate 2D filter banks. Our computational results show that the proposed method can be faster than (state-of-the-art) tensor Canonical Polyadic decomposition (CPD) method to obtain an approximation of comparable accuracy.