Mobile service robots often operate in human environments such as corridors, offices, classrooms, homes, etc. In order to function properly, they need to be aware of their 6 Degree of Freedom (6DoF) location. In addition, it is important that they possess semantic information i.e. knowledge of the types and positions of objects around them. In this method, we propose a method which obtains all of the above information directly. This method operates by using a camera as a 'semantic sensor The robot obtains the direction of objects such as doors, windows, tables, etc. around itself in 2D camera images by detecting bounding boxes. It then uses these object locations to localize itself within a floor map of the environment, which is typically available for most indoor environments. However, bounding box information is highly unstable due to the various changes in lighting, pose, size, etc. Hence, we also semantically tag feature points on detected objects and use them in our Monte-Carlo based localization framework. This increases the robustness and accuracy of our approach, as is demonstrated by experiments.