Schwartz was born on May 13, 1926 in Cheyenne, Wyoming. He attended Colorado Military School in Denver, Colorado and the University of Southern California. He left the university without a degree and went into the merchant tailoring business in Los Angeles, California. After that business went bankrupt, he moved to Las Vegas, Nevada with his brother, Seymour Schwartz, around 1951 and started Schwartz Brothers Clothing.