I always assumed the name was more of a reference to audio Master Recordings.

It’s the original tapes or whatever that all copies are derived from. It’s also where the term “remastering” comes from, as in “we went in and rebuilt a new master from the individual tracks, and this is the new master now”, versus just making another copy of the master for a re-release.