James Frey was born in Eureka, South Dakota, in 1941. His father worked in creamery and his mother was a registered nurse. When Frey was nine, the family, including his twin sister, relocated in Sioux Falls where his dad was plant manager for a dairy. He joined the YMCA in the fourth grade and ended up working for them until around the age of 22. He attended Augustana College in Sioux Falls, graduating with a major in sociology and a minor in history. After graduation, he worked for three years at the YMCA in Sioux Falls as program director.