Remote sensing mainly focuses on information extraction from data acquired by sensors on satellite and aerial platforms. Here, one such area of interest is ground object detection and shape extraction. Recently launched satellites and conventional aerial platforms (such as commercial UAV and professional drones) have sensors leading to more detailed and rich data source for this purpose. From these, data most of the times come in the form of optical images and LiDAR measurements. Resolution of this acquired data has increased significantly such that most ground objects (as buildings, trees, ships, cars, airplanes) can be detected and analyzed in detail. Therefore, computer vision methods have become extremely useful in remote sensing applications such as building detection and shape extraction for urban planning; tree crown measurement for crop yield forecasting; ship detection for monitoring unlawful fishery; car detection for traffic flow monitoring and intelligent transportation; and airplane detection for military and commercial operations. Researchers proposed several methods to automate the mentioned applications since manually handling them is extremely hard and prohibitively time consuming. Unfortunately, the proposed methods focus on one object type most of the times. Therefore, there is no general method to handle all the mentioned applications using computer vision tools. To overcome this problem, we propose a general framework for object detection and shape extraction in remote sensing data. Our method is based on probabilistic representation inspired by our previous work and perceptual organization principles. Due to space limitations, we only focus on buildings, trees, ships, airplanes, and cars as objects of interest in this study. We test the proposed method on several optical images acquired by different satellites and LiDAR data obtained from an aerial platform. For all objects of interest, we provide test results on both object detection and shape extraction steps. We analyze the proposed method based on these tests and discuss its strengths and weaknesses. We also comment on possible future extensions of the proposed method.