See Jerkstrand et al 2015 (Late time spectral line formation in Type IIb SNe), section 5.7.4.
One makes an assumption that O I 9226, 1.13 mu etc are recombination lines. One then gets a constraint on ne*f^(-1/2) psi^(-1/2), where psi is the fraction of electron provided by O II ions. In the models psi was typically between 0.5-1. In this formula temperature-dependency has also been abstracted away as its weak.
One proceeds by assuming that also Mg I 1.50 mu is a recombination line which is supported both by the models and by basic considerations. The models also show that magnesium almost always have most its mass as Mg II. This can be understood from the low ionization threshold of Mg I. One then takes advantage of this to translate n_MgII*V*f (appearing in the recombination formula) to M_Mg. Eventually the Mg I 1.50 mu luminosity will depend on M_Mg*ne.
Eliminating ne, a constraint on M_Mg f^-1/2 falls out. Using f ⇐ 1, an upper limit to the Mg mass can be set, but also taking f~0.1 (typical value inferred from other direction) the weak f dependency gives a good estimate of the Mg mass, not just the upper limit.