The viscoelastic properties of filaments and biopolymers play a crucial role in soft and biological materials from biopolymer networks to novel synthetic metamaterials. Colloidal particles with specific valency allow mimicking polymers and more complex molecular structures at the colloidal scale, offering direct observation of their internal degrees of freedom. Here, we elucidate the time-dependent viscoelastic response in the bending of isolated semi-flexible colloidal polymers, assembled from dipatch colloidal particles by reversible critical Casimir forces. By tuning the patch-patch interaction strength, we adjust the polymers' viscoelastic properties, and follow spontaneous bending modes and their relaxation directly on the particle level. We find that the elastic response is well described by that of a semiflexible rod with persistence length of order 1000 μm, tunable by the critical Casimir interaction strength. We identify the viscous relaxation on longer timescales to be due to internal friction, leading to a wavelength-independent relaxation time similar to single biopolymers, but in the colloidal case arising from the contact mechanics of the bonded patches. These tunable mechanical properties of assembled colloidal filaments open the door to “colloidal architectures”, rationally designed (network) structures with desired topology and mechanical properties.