I first came to Corton parish church at the end of the long summer of 2000. Over the previous eighteen months, I had visited every single church within a five mile stretch of the Suffolk coast, except this one. I had seen it often enough, its derelict tower a powerful ghost towards the sea from the A12, as that road thunders out of Norfolk. A couple of Suffolk parishes are further north; but Corton feels like the end of the world. Here, on the outermost tip of sprawling Lowestoft, among the holiday camps and waste disposal sites (why do they always put them so close together?) there is a feeling that we have stretched our sense of Suffolkness to an extreme. This, of course, is an illusion. Until 1974, the county boundary was a few miles further north, and the churches of Hopton, Belton, Fritton, Burgh Castle and Bradwell, now a suburb of Gorleston, were in Suffolk. Somehow, though, up here, especially in winter, the colours fade; the land and the sea are all too often wreathed in frets, the wide open vastness of Norfolk is but a step away, and I felt no sense of belonging.

The main road out of Lowestoft cuts across heathland, straight and undulating, with the wild sea off to the east. Suddenly, it narrows, and Corton is a surreal mix of pretty flint cottages and high fence-surrounded holiday camps, which I cycled hurriedly past, and then out again, for the church is beyond the village. St Bartholomew is haunting and lonely; there are many ruined churches along this coast, but this tower's gaping wounds, roughened by the salt sea winds, seem infinitely sombre and ancient. The tower dates from the 15th century, but the sea is only a few hundred yards away now, so I suppose it will not be here forever.

Corton is about as far as I can get from my house and still be in Suffolk. I came here in September 2010 with a sense of anticipation. I have been visiting this church on and off for about ten years, but so far I had never been able to get inside. At the time of my first visit there had been a notice with three keyholders on it, but not a single one of them was in, and so I headed back to Lowestoft in something approaching despair. I had really wanted to see inside. I had been told that the chancel had been restored in the 1870s as an Anglican parish church of appropriate size - for this was once a big church, and the nave, roofless and derelict only twenty years ago was now, apparently, a community hall. I looked through what was once the window to the west of the porch, and sawthat the area around where the font was once is now a pretty garden. But what I had really been looking forward to was a rare delight; a surviving medieval gable cross, with an image of Our Lady on one side.

I came back a few year later for the Suffolk Historic Churches bike ride. It struck me as a good time to visit the churches of Lowestoft again. Uniquely in Suffolk, the churches of Lowestoft are all kept locked; they appear to have a rather curious understanding of Christ's command for us to welcome the stranger at the gate around here. But theology goes out the window when there is money to be raised, and so I thought there was a fair chance that some of them at least would be taking part. In case you haven't heard of the Historic Churches bike ride, it is held on the second Saturday in September each year, and involves being sponsored to cycle around as many churches as possible. At each church you visit theysign you in, and they stamp your form, just to prove that you really did go there and didn't spend the day in the pub. The home church of the cyclist gets half the money raised, and so it is particularly popular with small rural churches. The event started in Suffolk nearly thirty years ago, and it has spread to almost every single other county in England, of which we are moderately proud.

I'd arrived that day at Corton at 9 o'clock, but not only was the church still locked, there was now no keyholder notice. I rang up the rector, and it turned out that there had been a mix-up with starting times. I waited for a full half-hour for someone to come and open the church, but nobody did. I couldn't wait any longer, and although the kind Rector's wife phoned me up later in the day to tell me that the church was now open, I was by then many miles away.

When I came back this way on the day of the 2010 Historic Churches bike ride, it was in the afternoon. I was on my way down from Great Yarmouth, planning to catch a train back to Ipswich from Lowestoft station. I turned the corner around the tower of St Bartholomew's church, and was pleased to see that, at last, the church was open. I am afraid that there is still no keyholder notice. But there was a charming man on duty who, despite recognising me abnd having read what I had previously written, was extremely courteous and helpful.

I stepped though the curiously low porch into the body of the nave. This has been divided into three parts. The western quarter is separated off with a glass screen, beyond which there is a pretty garden with the dramatic backdrop of the ruinous tower. Another wall separates off the eastern quarter, and in between is a large parish room, with a kitchen and toilets. Against the partition wall is a strange frieze composed of tracery and other rubble which once formed part of the church. The stonework seems to have been taken to a nearby farm for use in building, but was returned here in more recent years. Some of the tracery appears to be from the pedestal to the font.

A doorway in the south-eastern corner leads through into the body of the church. This consists of the truncated nave and a long chancel with a large perpendicular window filling the space with light. There are silvered bench ends set against the north wall, and the recut medieval font sits rather awkwardly in the middle of the nave; but otherwise the overwhelming feel is of the 19th century - and how could it not be, given that for many years this church was utterly ruinous? And yet this is a seemly, simple space, utterly suited to Anglican worship. I liked it a lot. The grand stone reredos appears to contain some medieval work, and at either end of it are the carved parish war memorials, one for soldiers and the other for sailors and fishermen.

There are no great memorials, and it seems that there were never any grand families leaving bequests at Corton. The great treasure of the church, the gable cross which I had longed to see ten years before, sits on a window sill with a mirror behind it so you can see both sides. There were once hundreds of these in Suffolk, but they were lost during the great Puritan destruction of 1644. Obviously the parish here was not Puritan enough - or perhaps they were just lazy. Whatever, it is the only one left in Suffolk.

The sculpture is of a beautiful Blessed Virgin and Child, and you can see at once what an extraordinary survival it was, and what treasures of English art we must have lost in our knee-jerk enthusiasm for puritanism. We live in an age when, if the Guardian message boards are anything to go by, such hysterical anti-Catholic fundamentism seems to be on the rise again; I looked up at the cross, and imagined the howling faces, the men with hammers. On the opposite wall is a simple memorial to William Allerton Soanes. He was, it tells us, lost at sea in the gale of May 28th 1860. "Watch therefore, for ye know not what hour your Lord doth come."