Ruby Duncan was born in Tallulah, Louisiana on June 7, 1932. Her parents passed away when Duncan was three years old and she spent the remainder of her youth living with various relatives in and around Tallulah. Duncan started work at the Ivory Plantation at an early age, only going to school part-time. She quit school to work full-time as a waitress and later a barmaid. Duncan left Tallulah for Las Vegas, Nevada in 1952 to live with her aunt near Henderson, Nevada.