A deliciously hot day in the summer of 2004; I am walking along the narrow lane over the hills from Bulmer with Jimmy, aged eleven, and Martha, who is seven. It is only a few miles, but we've brought drinks and sandwiches and they think it is a great adventure. Hardly any cars ever use this road, and so we can saunter and straggle safely. A couple of miles short of Kedington, we stop on the crest of the ridge for a picnic, and I gaze out across the haze of the valley.

Harvest dust hangs heavy in the air like smoke, and Kedington is invisible in the valley below. Nothing stirs; there is a stillness in the heat, and it seems impossible that we are less than fifty miles from the centre of London. Here, right on the edge of it, is one of the most intensely rural places in East Anglia.

And yet, we are hardly any distance from one of the region's most derided towns. In an earlier version of this entry, I referred to Kedington as a suburb of Haverhill. This didn't go down terribly well with the natives, I can tell you. Haverhill is an unloved place, especially by those in the surrounding villages who see it encroaching over the hills. In fact, the sprawl of Haverhill is into neighbouring counties, Essex and Cambridgeshire, which are presumably more used to that kind of thing than sleepy Suffolk. Up here on the ridge, I could turn and see the industrial wasteland of south Haverhill like a shining city on the horizon.

In our modern world, the nature of suburbia has changed. It needn't be physically joined to the towns and cities it serves, and I think that there is something suburban about Kedington, but in a good way. Here are the pleasant houses, the school, a shop or two, a sense of being on the outskirts of a busy place. And it isn't just Haverhill; as well as the jobs in town, there is Stourmead Hospital in the village, and food processing factories to the north of it. This is not a dying part of rural England.

If Kedington is busy, it still focuses on its church, as it has done for a millennium. This is one of the most interesting and significant churches in all East Anglia,and one of the most welcoming. It is as if the church has set out to be the heart of the community, and succeeded.

Some East Anglian churches are all of a piece, and are famous for it; think of Lavenham, Salle and Southwold. But some churches are a ramshackle accumulation of centuries of care and neglect; the patching up, mending and making do of generations. Such churches are unique treasure troves of local and national history, and Kedington is one of them.

Your first impression of St Peter and St Paul will probably be quite how battered and long it is, like a scruffy lizard lying out in the sun. It hugs the earth; this is accentuated by the way that the entire roof ridge of the nave appears to have been scraped off, as if by a giant knife cutting through butter. Running along the top of the roof are three large lids. Can they be sky lights? The rendered chancel appears to have come from another church altogether. At the other end, the gawky porch and solid tower also appear as if they have been bolted on as afterthoughts. And what on earth is the red brick quatrefoil offset on the eastern face of the tower? An extraordinary building then, and quite unlike any other.

Even if you couldn't go inside, the exterior would be worth coming to see. But fortunately, this is a welcoming church, open to visitors every day. You step inside to quiet music playing, and look up to see that you were right; they are skylights. They backlight the late medieval roof beams strangely and beautifully. They were installed in the 1860s by someone who thought it would be a good idea; perhaps he was laughed out of the parish, because the Victorians appear to have done very little else here.

The great range of benches in the nave is 15th century, but the overwhelming feel is of the 17th and 18th centuries. Age-bleached box pews are shoe-horned into the aisles, and there are banked seats on both sides at the west end. These were for the boy and girl scholars of the parish, and you'll notice that the most westerly box pew in each aisle has a seat facing westwards, so that the master and mistress could keep an eye on their charges.

At the east end is a very fine triple-decker pulpit dated 1610, but even this is dominated by the extraordinary Barnadiston family pew, as big as a bus. The Barnadistons dominate this parish as the pew does the church, but it is most interesting for the fact that it is constructed from the panels and tracery of the medieval roodscreen. The lower part retains its original colour, and there are lions, dragons and eagles in the spandrils.

It is a testimony to the power of this building that it contains more than twenty massive and in some cases overbearing monuments to the Barnadistons, and yet not once does it feel like a mausoleum. Indeed, it doesn't really feel like a Big House church; perhpas it is the box pews, but it is easier to imagine the wheelwright and the ploughboy sitting in this church than the Barnadistons. In the family pew, they must have felt liked caged animals. That said, there are some remarkable memorials here, including effigies, cherubs, angels and at least a dozen skulls. The juxtaposition between the largest, at the east end of the south aisle, and on either side of the theological divide of the Reformation, is quite moving.

If there is a contrast between the chancel and the nave on the outside, it is doubly so once you get inside. As if to make the point, here is a rare 1619 folding roodscreen, and beyond a wide, light chancel space under a low-pitched roof, with its original set of three-sided Laudian altar rails still intact.

We are spoiled in Suffolk, where half a dozen examples survive; there is just one in Norfolk, and they are extremely rare in a country church. As if the whole church had been drawing you to this moment, an early 20th century reredos set in Jacobean panelling depicts the Blessed Virgin and the infant Christ. immediately above is a circular crucifix rescued from the churchyard. It is Saxon, at least a thousand years old, from the original Christian settlement on this site, and I defy you not to be moved.

St Peter and St Paul, Kedington, is in the centre of the village just to the south of the main Haverhill to Bury road. You can also reach it from the Haverhill to Colchester road. It is open to visitors every day.

this site supported by commission from amazon.co.uk