In this paper, we introduce a stochastic generation framework (SDVI) to infill long intervals in video sequences. To enhance the temporal resolution, video interpolation aims to produce transitional frames for a short interval between every two frames. Video Infilling, however, aims to complete long intervals in a video sequence. Our framework models the infilling as a constrained stochastic generation process and sequentially samples dynamics from the inferred distribution. SDVI consists of two parts: (1) a bi-directional constraint propagation to guarantee the spatial-temporal coherency among frames, (2) a stochastic sampling process to generate dynamics from the inferred distributions. Experimental results show that SDVI can generate clear and varied sequences. Moreover, motions in the generated sequence are realistic and able to transfer smoothly from the referenced start frame to the end frame.