Growing up in St. George's near the Chesapeake and Delaware Canal in the late 1800s, Ed Fields was part of a thriving African-American community. His mother and father worked for white families in St. George's and Odessa, and they lived for a while in town, then on a farm. The children attended school in a building that also served as the church, and which had been sold to the black community when St. George's built a new school. Fields observed white residents enjoying sports such as fox hunting and ice skating. Hundreds from nearby towns such as Chesapeake City, Summit Bridge and Delaware City would flock to a nearby pond to skate and drive on the ice in horse-drawn sleighs. Afraid of the ice himself, Fields would stand on the wharf and "keep a great big fire for them to get warm by."

Fields fondly remembered the simple entertainments he and his friends devised as children

Fields recalled that "I just tell you, to my mind I think we had a better time than the children do today, 'cause somebody always has to spend a lot of money to give things to their children. Our parents didn't have it, so what few pennies we had, we enjoyed it just as well, more so than they do today." Eventually, most of St. George's black residents moved to nearby towns, such as Delaware City, where they could find better work.