The human phallus stops growing around the ages of 16-17, and its size is likely due to heritable factors. By what physiological mechanism does the penis stop growing, given that there are still elevated levels of androgens (the growth factor for erectile tissue) in the adult male, and that androgen receptors don't downregulate until several years after the penis stops growing?

correct.

Penile size in an individual and when it stops growing will be determined by many genetic (including various transcription factors) and sex hormonal determinants. the specifics of both are still largely unknown and are surprisingly poorly studied.