A novel video-based method is proposed for long-distance wildfire smoke detection. Since the long-distance wildfire smoke usually moves slowly and lacks salient features in the video, the detection is still a challenging problem. Unlike many traditional video-based methods that usually rely on the smoke color or motion for initial smoke region segmentation, we use the Maximally Stable Extremal Region (MSER) detection method to extract local extremal regions of the smoke. This makes the initial segmentation of possible smoke region less dependent on the motion and color information. Potential smoke regions are then selected from all the possible regions by using some static visual features of the smoke, helping to eliminate the non-smoke regions as many as possible. Once a potential smoke region is found, we keep tracking it by searching the best-matched extremal regions in the subsequent frames. At the same time, the propagating motions of the potential smoke region are monitored based on a novel cumulated region approach, which can be effectively used to identify the distinctive expanding and rising motions of smoke. This approach can also make the final smoke motion identification insensitive to image shaking. It was proved that the proposed method is able to reliably detect the long-distance wildfire smoke and simultaneously produce very few false alarms in actual applications.