In order to improve the efficiency of coal mining, accelerate the backfilling speed of goaf and reduce the pollution and damage of coal-based solid waste to mining environment, coal gangue-based paste filling materials with fast hardening characteristics are prepared using coal gangue, fly ash, sulphoaluminate cement and persulfate cement as raw materials. The influences of aggregate-cement ratio and water-cement ratio on the macroscopic properties of the specimens are studied through a series of characterization tests such as fluidity, unconfined compressive strength (UCS), water absorption, softening coefficient and so on. The microstructure and curing mechanism of the specimens are analyzed by TG/DTA, XRD and SEM. The results show that when the aggregate-cement ratio is 4:1 and the water-cement ratio is 1.5, the initial fluidity of the specimen is 195 mm, and the 8 h UCS is 3.26 MPa, displaying obvious characteristics of early strength and fast hardening, and the UCS at 3, 7, and 28 d are 5.56, 5.66, and 6.61 MPa, respectively. The water absorption rate of the sample at 28 d age is 16.86%, and the softening coefficient is 0.90, indicating its excellent water resistance. The phase composition and micro-morphology analyses show that the early unconfined compressive strength of the filling specimens mainly comes from the accumulation and filling benefits of the gangue aggregate as well as the formation of a large amount of ettringite (AFt) in the cementation part. Moreover, the raw materials such as fly ash induce a pozzolanic reaction and react synergistically with persulfate cement, generating a certain amount of hydration products in the later period of curing, which further improves the densification and mechanical properties of the filling specimens. The results obtained in our current research can provide experimental basis and theoretical guidance for the development and practical application of new coal gangue-based green filling materials.