I first came to Corton parish church at the end of the long summer of 2000. Over the previous eighteen months, I had visited every single church within a five mile stretch of the Suffolk coast, except this one. I had seen it often enough, its derelict tower a powerful ghost towards the sea from the A12, as that road thunders out of Norfolk. A couple of Suffolk parishes are further north; but Corton feels like the end of the world. Here, on the outermost tip of sprawling Lowestoft, among the holiday camps and waste disposal sites (why do they always put them so close together?) there is a feeling that we have stretched our sense of Suffolkness to an extreme. This, of course, is an illusion. Until 1974, the county boundary was a few miles further north, and the churches of Hopton, Belton, Fritton, Burgh Castle and Bradwell, now a suburb of Gorleston, were in Suffolk. Somehow, though, up here, especially in winter, the colours fade; the land and the sea are all too often wreathed in frets, the wide open vastness of Norfolk is but a step away, and I felt no sense of belonging.

The main road out of Lowestoft cuts across heathland, straight and undulating, with the wild sea off to the east. Suddenly, it narrows, and Corton is a surreal mix of pretty flint cottages and high fence-surrounded holiday camps, which I cycled hurriedly past, and then out again, for the church is beyond the village. St Bartholomew is haunting and lonely; there are many ruined churches along this coast, but this tower's gaping wounds, roughened by the salt sea winds, seem infinitely sombre and ancient. The tower dates from the 15th century, but the sea is only a few hundred yards away now, so I suppose it will not be here forever.

Corton is about as far as I can get from my house and still be in Suffolk. Uniquely in Suffolk, the churches of Lowestoft are mostly kept locked, and so I came here on the day of the Historic Churches Bike Ride of September 2010 with a sense of anticipation. I had been told that the chancel had been restored in the 1870s as an Anglican parish church of appropriate size - for this was once a big church, and the nave, roofless and derelict only twenty years ago was now, apparently, a community hall. I looked through what was once the window to the west of the porch, and sawthat the area around where the font was once is now a pretty garden. But what I had really been looking forward to was a rare delight; a surviving medieval gable cross, with an image of Our Lady on one side.

I stepped though the curiously low porch into the body of the nave. This has been divided into three parts. The western quarter is separated off with a glass screen, beyond which there is a pretty garden with the dramatic backdrop of the ruinous tower. Another wall separates off the eastern quarter, and in between is a large parish room, with a kitchen and toilets. Against the partition wall is a strange frieze composed of tracery and other rubble which once formed part of the church. The stonework seems to have been taken to a nearby farm for use in building, but was returned here in more recent years. Some of the tracery appears to be from the pedestal to the font.

A doorway in the south-eastern corner leads through into the body of the church. This consists of the truncated nave and a long chancel with a large perpendicular window filling the space with light. There are silvered bench ends set against the north wall, and the recut medieval font sits rather awkwardly in the middle of the nave; but otherwise the overwhelming feel is of the 19th century - and how could it not be, given that for many years this church was utterly ruinous? And yet this is a seemly, simple space, utterly suited to Anglican worship. I liked it a lot. The grand stone reredos appears to contain some medieval work, and at either end of it are the carved parish war memorials, one for soldiers and the other for sailors and fishermen.

There are no great memorials, and it seems that there were never any grand families leaving bequests at Corton. The great treasure of the church, the gable cross which I had longed to see ten years before, sits on a window sill with a mirror behind it so you can see both sides. There were once hundreds of these in Suffolk, but they were lost during the great Puritan destruction of 1644. Obviously the parish here was not Puritan enough - or perhaps they were just lazy. Whatever, it is the only one left in Suffolk.

The sculpture is of a beautiful Blessed Virgin and Child, and you can see at once what an extraordinary survival it was, and what treasures of English art we must have lost in our knee-jerk enthusiasm for puritanism. We live in an age when such hysterical fundamentalism seems to be on the rise again. I looked up at the cross, and imagined the howling faces, the men with hammers. On the opposite wall is a simple memorial to William Allerton Soanes. He was, it tells us, lost at sea in the gale of May 28th 1860. "Watch therefore, for ye know not what hour your Lord doth come."