Phosphogypsum is a by-product generated during the wet-process production of phosphoric acid, with enormous annual output. The stockpiling of large quantities of phosphogypsum not only occupies considerable land resources but may also cause environmental and safety issues such as heavy metal leaching, soil salinization-alkalization, and dump collapses. The resource utilization is therefore urgently needed. At present, phosphogypsum is mainly applied in agriculture, industry, and the construction industry. Due to its complex composition, the utilization is challenging owing to the high pretreatment cost. Based on the concept of waste recycling, this study uses phosphogypsum as the main raw material, supplemented by ground granulated blast furnace slag, fly ash, and type II anhydrite. Two foaming agents, sodium bicarbonate and aluminum powder, were adopted to regulate the pore structure, and the effects of different foaming agents on the performance of ceramsites are compared and analyzed. The sample with the highest cylinder compressive strength is further characterized by XRD, SEM, and BET to investigate its micromechanism. The results demonstrate that the type of foaming agent significantly influences the porosity, bulk density, and cylinder compressive strength of ceramsites. Under identical preparation conditions, the 7 d cylinder compressive strength of ceramsites prepared with aluminum powder is higher than that prepared with sodium bicarbonate. With appropriate raw material proportions and a suitable foaming agent, the non-fired ceramsite achieves a maximum cylinder compressive strength of 6.5 MPa, which basically meets the strength requirements for aggregates in lightweight aggregate concrete. This study realizes the green preparation of non-fired ceramsite with favorable environmental and economic benefits, and provides a new way for the large-scale utilization of phosphogypsum.