James Frey was born in Eureka, South Dakota, in 1941. His father worked in creamery and his mother was a registered nurse. When Frey was nine, the family, including his twin sister, relocated in Sioux Falls where his dad was plant manager for a dairy. He joined the YMCA in the fourth grade and ended up working for them until around the age of 22. He attended Augustana College in Sioux Falls, graduating with a major in sociology and a minor in history. After graduation, he worked for three years at the YMCA in Sioux Falls as program director.

A street on the Westside is named for Elgin Holbert's grandmother, Viola Cunningham, who was an early land owner. It is believed that in 2002 she donated the property for Madison School now renamed Wendell P. Williams Elementary School. Although from Eudora, Arkansas, a few miles from Mississippi, his parents are a mixed couple, mother is White and father, Black. His mother was treated well in the Westside community but was very private concentrating on rearing her children with little community interaction.