In single-molecule localization microscopy (SMLM), the use of engineered point spread functions (PSFs) provides access to three-dimensional localization information. The conventional approach of fitting PSFs with a single 2-dimensional Gaussian profile, however, often falls short in analyzing complex PSFs created by placing phase masks, deformable mirrors or spatial light modulators in the optical detection pathway. Here, we describe the integration of PSF modalities known as double-helix, saddle-point or tetra-pod into the phasor-based SMLM (pSMLM) framework enabling fast CPU based localization of single-molecule emitters with sub-pixel accuracy in three dimensions. For the double-helix PSF, pSMLM identifies the two individual lobes and uses their relative rotation for obtaining z-resolved localizations. For the analysis of saddle-point or tetra-pod PSFs, we present a novel phasor-based deconvolution approach entitled circular-tangent pSMLM. Saddle-point PSFs were experimentally realized by placing a deformable mirror in the Fourier plane and modulating the incoming wavefront with specific Zernike modes. Our pSMLM software package delivers similar precision and recall rates to the best-in-class software package (SMAP) at signal-to-noise ratios typical for organic fluorophores and achieves localization rates of up to 15 kHz (double-helix) and 250 kHz (saddle-point/tetra-pod) on a standard CPU. We further integrated pSMLM into an existing software package (SMALL-LABS) suitable for single-particle imaging and tracking in environments with obscuring backgrounds. Taken together, we provide a powerful hardware and software environment for advanced single-molecule studies.