Autonomous driving has been powered by the current success in deep learning. However appearance information is insufficient for that end goal. This motivated Our Idea to utilize Motion and Geometric Priors with Deep Learning. One application which we show this is to perform real-time segmentation based on these priors. A need for such an idea can be incorporated in many driving assisted systems such as rear cross traffic alerts. That indicates to the driver the existence of moving objects in the driver’s blind spot. It can also be used for detecting moving objects from aerial vehicles that are harder to obtain with off the shelf detectors. Our framework is built using Tensorflow and is deployed on Nvidia Jetson Tx2.