Over the last couple of centuries, the Borough of Lowestoft has sprawled out to encompass a number of historic parishes, including Gunton, Pakefield, Oulton, Carlton Colville and here, the former village of Kirkley. Consequently, seven of the churches in the Borough are medieval - or, more strictly, of medieval foundation, for there was a spectacular 19th Century restoration here at St Peter and St John, which has left very little of medieval origin. The church, then simply dedicated to St Peter, had been abandoned in 1680, after more than a century of neglect. The parishioners moved in up the road at Pakefield, but when the two parishes there were combined, and the dividing wall was opened up, there was no longer room for them. So they came back, patched up this church, and suffered in it for the next 120 years. However, by the 1870s, when Lowestoft had expanded and engulfed the parish, the time had come for a total rebuilding. Another church, the now demolished St John, had been built in the parish ten years before. But this was not enough for the huge population, so St Peter was built anew between 1874 and 1876 by the architect John Clemence. All that survives of the medieval church is the heavily restored tower, which once stood at the west end, but the new building was extended southwards, putting it at the north-west corner.

From the north, it looks conventional enough. But on the south west side there is a delightful surprise; an apsidal baptistery, the work of Thomas Porter in 1893. This is quite unfamiliar on an Anglican church in Suffolk, although there is something similar at St Edmund Catholic church in Bungay.

Like all churches, Kirkley Parish church continues to reinvent itself. When St John was demolished in 1976, it took on the current dual dedication. These days, the name Kirkley is roughly synonymous with that part of coastal Lowestoft south of the river, which isn't entirely accurate. The main road to Ipswich is a ratrun through the parish, giving it an urban scale beyond its size, as if it was some little-known district of London. Off this parade of anonymous grocers, newsagents and kebab shops you find yourself in a pleasant residential suburb. Towards the sea, the houses are enormous. This area, Kirkley Cliffs, is a rather pleasant seaside resort, usually referred to as Lowestoft South Beach. Until quite recently this was a fairly seedy area, but it has undergone a huge improvement programme in the last ten years, thanks to an injection of money from the European Social Fund for deprived areas. Before coming here, I had been chatting with the man at South Lowestoft Methodist Church about this. "A lot of local people think it was a bit of a waste of money", he observed. Oh, believe me, I wanted to say, remembering how it had been before, it was really, really worth it. And now, Lowestoft's are the only beaches in Suffolk to win Blue Flag awards, and are of a beautiful golden sand; why people would rather go ten miles up the road to seedy, be-littered Great Yarmouth beats me.

In one of the large houses, now a guesthouse but until recently a dentist's surgery, as it had been for nearly a hundred years, one of the most significant events in Kirkley's history occured. In this house, on November 22nd 1913, a baby boy was born to the wife of the dentist of the day. This boy, their youngest child, would grow up to be the most famous Suffolker of the 20th century; and it is entirely appropriate that Benjamin Britten should be born on St Cecilia's Day, for she is the patron saint of music.

Britten is more closely associated with Aldeburgh, Snape and Orford than Kirkley, not least because of the music which was first performed in the churches there. But it was at the High Church St Peter and neighbouring St John that Britten first experienced the power of the liturgy, and church music first entered his consciousness. There is a blue plaque on the house, but no other fuss seems to be made of Britten in Kirkley; you have to search to find him. At the northern end of Lowestoft is the Benjamin Britten High School, and, more surreally, the Benjamin Britten Shopping Centre is in the centre of town. But Kirkley seems rather reserved about its genius son, who was certainly the greatest English composer of the 20th century, and the finest English composer of opera ever. He was also, perhaps, the first establishment gay figure to openly win acceptance, and he is buried today beside his partner for 40 years, Peter Pears, in the graveyard at Aldeburgh.

The first time I came here, shortly after the start of the new millennium, I walked around St Peter and St John on its little hill, surrounded by pleasant little terraced houses. They must have been built at about the same time. The church looks abandoned from this aspect, but I took this to be merely another manifestation of the urban shabbiness that Lowestoft in from the coast seems to rejoice in. In fact, the noticeboard at the east end, facing Kirkley High Street, shows that this is a lively parish.

However, the church was locked. I stood in the scrubby churchyard, feeling a bit fed up, but I was cheered immensely by being recognised by a man walking through. "You're that bloke who's visiting all the churches, aren't you?" he said. "What do you think of this one?"

Having just read Sam Mortlock's account, I called it to mind, and gave him a brief guided tour of the outside. He thought for a moment. "So, you're saying it's not as old as it looks?" He seemed perplexed.

"That's right", I assented.

"Oh," he said. "Ah, well." And so, we went our separate ways, him reflectively towards the shops, me feeling slightly guilty for having disappointed him.

It wasn't for almost another ten years that I came back to Kirkley and tried to get inside again. Actually, I did not have try very hard, because I visited on the day of the Historic Churches Bike Ride 2009. It is worth adding that, having taken part in this event for more than twenty years, I have a wide experience of the kind of people who sit and welcome cyclists to churches on this day. I can tell you, in all honesty, that the people of Lowestoft are probably the friendliest people I have ever had any dealings with - and not just the church welcomers. In shops and pubs, on buses and trains, even when just walking down the street, these are the kindest, most courteous people you could possibly hope to meet. And yet... and yet. When you turn up at their churches and want to see inside, they look at you as if you had requested something really off the wall, like suggesting you wanted to examine the contents of their wallets, or enquired as to the colour of their underwear. Even on Bike Ride Day I was treated with mild surprise and some raised eyebrows when I asked to step past the welcome desk into the body of the church. No one actually refused, but you could tell that they thought I was a little odd.

I suppose it is a cultural thing. Most parishes outside Lowestoft seem to see their building as an act of witness in itself, and that by keeping it open they offer a prayerful space in the middle of a busy world to passing pilgrims and strangers. Evangelical churches are, not unreasonably, less likely to take this approach - they want you to come on a Sunday and join the Club, after all - but Kirkley parish church is very much in the High Church tradition. I suspect that they simply do not know how things are arranged elsewhere. In rewriting this, I have been tempted to remove an earlier postscript which detailed Peter Stephens' energetic and yet ultimately fruitless attempt to borrow the key here. However, as the church is still kept locked, and the same Rector is in charge, and in any case scores of people have written to me to say it is one of the funniest things they have read on the site, I have retained it below.

For myself, it was a delight to finally set foot inside St Peter and St John. It is a big, urban, late 19th Century Anglo-catholic space, with white- and red-brick banding creating a polychromatic effect, a candle flickering in front of the statue of Our Lady of Walsingham by the south doorway, and jewel-like windows piercing through the gloom. The best is in the east window, good Kempe glass depicting Christ in Majesty flanked by the four archangels Michael, Raphael, Gabriel and Jophiel.

To get to it, you step through the excellent wrought-iron rood screen of the 1890s. On the north side of the chancel is the organ where Benjamin Britten's mother came to practice on weekday afternoons, the small boy sitting beside her and turning the pages. It is apparently one of the few four manual organs in Suffolk.

There is, of course, no tower arch, and a great clock ticks steadily at the west end of the nave. The Ward & Hughes glass down the south aisle is not the workshop's best, but contributes to the feel of a faded, early 20th century Anglo-catholicism which is being lost to us by reorderings and redecorations elsewhere. One set forms the war memorial window, a touching moment. At the west end of this aisle is the church's most memorable feature, the open baptistery with its wrought iron screenwork and spectacular marble font. I did try to imagine the infant Britten being dipped into it, but I suspect that he was actually baptised at the now-demolished St John. The nice lady on duty, seeing me gazing at the font, came and apologized for its dusty state. "The whole church has got so dirty, and it needs a thorough redecoration. But it is so expensive, and there are so few of us left now." I told her that I rather liked it the way it was, but I suspect that was small comfort.

One intriguing survival is the small reredos made by Robert Anning Bell for St Peter's church in north Lowestoft, which was demolished, along with St John, in the 1970s. The rationalisation of Lowestoft's Anglican churches at that time bolstered congregations in the churches which survived. It is sad to think that the Diocese of Norwich may very well need to instigate the same process again in the not too distant future. One can only hope that Kirkley will not be one of the victims.

Postscript from 2002: Peter Stephens writes: I was also unable to get into Kirkley. The Rector is a very strange man! Even by the time I got home I was still wondering what was the matter with him. He was very friendly and chatty, smiled the whole time I was talking to him, but he just would not go and get the key. I first noticed him quietly trimming a hedge in the garden of the house next to the church. Finding the church locked - what a surprise! - I sauntered over to him and said, ' I gather this is the place to come to for the key to the church.'

'Ah, well, no, not really,' he said.

'It says 'Rectory' on the gate.'

'Yes, but it's not strictly so.'

'Do you mean it's a private house now?'

'Oh, no.' He paued, then rather reluctantly said, 'You see, well, I'm the Rector actually.'

'Good. So could I please borrow the key?'

'Well, no, it's my day off.'

'But surely I can just borrow the key?'

'I'm afraid I've got to get this hedge cut.' He observed me studying the small amount of hedge he had to cut and said, 'If you come back on another day I'll happily make you a cup of tea.'

I felt like saying, I don't want a cup of tea - I want to see inside the church. Now what's the problem? I said, 'I don't live round here.'

'Oh, what a pity! Where have you come from?'

'Norwich.'

'You're just here for the day then?'

Yes.'

'Well, do have a lovely day.'

'I'm trying to. But I'd like to see inside the church.'

'Of course. You could go down to the High Street - no, he's closed today. And both the churchwardens are at work - well, not work as such. They're away.'

'But surely you have a key?'

'Well yes. But why don't you give me a ring and, as I said, I'll give you a cup of tea?'

What is this with the cup of tea? I almost blurted out. I didn't want a cup of tea. I wanted to see inside the church. But I remained cool. He was, as I said, very friendly and chatty and the dialogue was much longer than I've suggested here, but mostly repetitive. I tried a different approach. I thought I might appeal to his better nature through romantic association.

'I got married in Lowestoft,' I said.

'Really?'

'Forty years ago tomorrow.'

'Well, that's very good. Do have a lovely day, and let me offer you my congratulations. Which church?'

'Well, we'd have liked Blythburgh, but certain difficulties in the life of the bride's father at the time regrettably had us land up at the Registry Office.'

'Oh, nothing regrettable in that. Forty years is a long time. So many big church weddings end up with the couple separating in less than a year sometimes.'

'Absolutely. But my problem is that it's difficult for me to get to Lowestoft. Couldn't I just see inside the church now?'

'I'm afraid I really do have to get this hedge cut.' I gave it up. Looking at my watch I said, 'Well, I see I've already detained you from it for twenty minutes,' leaving him to draw the conclusion that it would have taken him about two to step inside the house and get the key. And what did he say? 'Do give me a ring next time you're this way and, as I said, I'll be delighted to make you a cup of tea.' I departed on the verge of screaming.