Naphthalene diimide (NDI)-based conjugated polymers are prominent polymer acceptors in all-polymer solar cells (all-PSCs). However, these polymers tend to exhibit strong self-aggregation, which can cause excessive phase separation and hinder optimal donor-acceptor mixing in the bulk heterojunction blend. To address this issue, random terpolymerization was employed to modulate the aggregation of NDI-based polymer acceptors, aiming to enhance the corresponding device performance of all-PSCs. Four terpolymers (PNDI-T21, PNDI-T23, PNDI-T25, and PNDI-RT) were synthesized by incorporating 10 mol % thiophene derivatives into PNDI-T2, a reference polymer with a regular configuration. Increased thiophene content enhanced backbone planarity, leading to greater aggregation and crystallinity, while a highly randomized backbone reduced both. When used as polymer acceptors in all-PSCs, PNDI-T21, with the weakest aggregation, achieved the highest power conversion efficiency (5.3%), whereas PNDI-T25, with the strongest aggregation, showed the lowest efficiency (3.2%).