This article focuses on the development of tools for automatic recognition and segmentation of the main geometrical characteristics of heritage buildings (walls, towers, roofs, slopes, etc.) to simplify a 3D point cloud into simpler model based on geometric primitives. After evaluation of well known techniques for point cloud segmentation, an hybrid method based on region growing algorithm and primitive fitting by Sample Consensus appears as the most successful. Then, a refinement process is applied by grouping close-by points into voxels and assigning them to the closest primitive. The final algorithm is tested in the front wall of the castle of Chambord, France showing a 94.40% coincidence between the geometric primitives found and manual ground truth. This algorithm might prove useful for obtaining simpler models of cultural heritage structures, which can be used for storage, manipulation and even other types of analysis such as finite element models.