Why do we say that healing a relationship is “mending fences”?

In 1880, the strong-willed senator John Sherman was testing the water for a presidential nomination. He slipped out of Washington but was followed to his Ohio farm by a reporter who found the senator talking with a high-ranking party official while standing near a fence.

When the reporter asked what they were doing, the response, “We’re mending fences,” gave him his headline, and it became a new phrase for healing relationships. Within a democracy, what are the fourth and fifth estates?