I do not think I had ever seen so many hares before, but they were in every field as I headed south-east out of Bury St Edmunds up the rambling, hilly lane to Rushbrooke. I'd seen half a dozen before I was even a mile from the turn-off. Two of them were having fisticuffs near the pig fields, and I hauled my bike off into the verge to watch. I must have stood there for twenty minutes, gazing through the hedgerow, and they must have known I was there - the pigs certainly did. But spring madness had infected them, and they had other priorities than worrying about me.

I got even closer to one old fellow. He was as big as a dog, and lay sullenly in the furrow of a ploughed hillside barely twenty feet from the road. Perhaps he was sulking that, at his age, he was missing out on all the excitement. Eventually he lifted himself out of the rut, and hauled himself up the bank, his powerful legs kicking back slowly behind him, as if he owned the place. Which, of course, he did.

It was the third time I had made this journey over the last two months, and it was pleasing to see the way the landscape was changing. The first time had been on a day in early February, and I had been shocked to find Rushbrooke church locked; in more than half a dozen visits previously, I had always found it open, and so had other people who I asked. There was a keyholder notice, but the keyholder was out.

I wandered about the graveyard sadly, taking photographs of snowdrops. When I got home, I rang the Rector, who lives at Rougham, to ask what was happening. It seemed that it was too difficult now to find anyone to open and shut the church everyday. It was still open every Saturday, but he gave me the phone number of the keyholder, adding hurriedly that he didn't have a key himself.

The keyholder was very happy to help me, and seemed delighted that anyone should be interested. We arranged that the church would be left open for me, and so on a Thursday in half term my daughter and I cycled up through the steep countryside again. It took longer this time, of course, but it was the first really warm, sunny day of the year, and all the birds seemed to know it. There were hardly any cars about - I think we saw only one between leaving Bury and reaching the Sudbury road from Little Whelnetham some four miles later - and I was glad to be there.

But when we got into the church, disaster struck. After a few shots, my camera batteries ran out. I reached in my bag for new ones, but I had forgotten to pack them. There are, of course, no shops in Rushbrooke. If it had been me on my own, I would have sighed, shrugged, and headed the two miles back into Bury to get some, but I did not want to spoil Martha's bike ride, and so after a wistful look around, we headed on to Bradfield St George.

And so, in April, with the lunatic hares oblivious to the way they lifted my heart, I came to Rushbrooke again. If you like something a little out of the ordinary, then Rushbrooke is exactly the kind of place you'd find interesting. On its ridge above the valley, this little settlement is quite unlike any other. Rushbrooke Hall, home to generations of the Jermyn family, was Suffolk's largest and finest moated Tudor mansion. Used for housing troops during the war, it suffered a mysterious fire, and was demolished without permission in 1961. Pevsner called it a capital loss, a tragic disruption of the post-war Suffolk landscape.

There is a brick well house at a turn in the lane, with some extraordinarily good farmworkers' cottages beyond. They were built in the early 1960s, by Llewelyn-Davies and Weekes. St Nicholas is just beyond, looking reassuringly familiar, its 14th century tower and 15th century everything else all plastered. Mortlock tells us that, beneath this skin, the tower is flint, but the rest brick. Indeed, brick surrounds are evident on several windows. There are fine headstops on the west window, and on the windows of the south aisle.

If I tell you that this church was extensively restored in the 19th century, you might imagine a well-polished but anonymous interior, all Minton tiles and deal benches. You'd expect recut stonework and a garish reredos. It would all be pleasantly ordered for late 19th century sacramentalist worship, rather out of date now but obviously cared for, as at hundreds of other Suffolk village churches.

If you thought that, then you would be in for a mighty surprise. For this is perhaps the most extraordinary of all Suffolk church interiors. It is the work of, and a testimony to, one of the great Suffolk eccentrics. The first time you realise that something decidedly odd has taken place is when you step from the porch into what you believe to be the south aisle. Instead, you find yourself in a most unusual vestibule; to the east, your right, is a solid partition wall. A few feet ahead of you is another partition, less substantial, and not going all the way up to the ceiling. The intention seems to be a baptistery, and there are two fonts, a recut stone medieval one and a wooden Victorian one.

I'm not sure where the stone one came from; it was brought here in the 1980s, and reunited with its column. It may even have been the original font, removed in the 19th century and returned here. But the wooden font is more interesting; an idiosyncratic design, the work of Colonel Rushbrooke, who lived at the Hall in the early 19th century. It is his monument low down on the south wall here.

If you have already visited nearby Nowton, you will have seen there the 84 roundels of Flemish glass plundered by the Rushbrookes from Belgian monasteries. Here, Colonel Rushbrooke recycled panelling from the Hall and other sources to create what can only be described as a Gothick Fantasy. For the best first impression, ignore the entrance into the main part of the church ahead of you, turn left, go past the fonts and enter it from the west. As you pull aside the curtain and step in, you may be unbalanced slightly for a moment by the sheer lack of familiarity. Rushbrooke, inspired perhaps by happy memories of his youth, recreated here a college chapel quire, along the lines of Peterhouse, Cambridge. Banked dark seats face inwards, awaiting choral scholars.

At the west end, there is a dramatic array of organ pipes. They are an elaborate fantasy, a conceit connected to no instrument, purely for decoration. turning east, the tympanum is still in place, and on it is something even more remarkable, which we'll return to in a moment.

Beyond, the chancel is full of light, a balance to the serious gloom of the quire. Medieval glass is offset by blue surrounding panes, and the banners of Jermyns and Rushbrookes hang down in front. If you turn southwards, you see that the rest of the aisle is divided into two. The middle part is now a vestry, but the eastern range is a funerary chapel to the Jermyns. Of several monuments, the best is to Thomas, the last of the Jermyns. He died in 1692, in a boating accident on the River Thames. He was just 15 years old. A mast collapsed, and landed on his head; and so, after centuries, a great landed family became extinct.

Colonel Rushbrooke's vision may seem ludicrous if you just read about it, but the thing is, it undoubtedly works; it's a splendid interior. He must have been dead proud of it. And yet, it wasn't quite triumphalist enough, and did not articulate sufficiently the Establishment of the Church. And so, on the tympanum at the east end of the nave, he put in place the great royal arms of Henry VIII, the only Henrician set of arms in all England.

Various claims have been made that the arms are, in fact, genuine. The church guidebook still insists on this, pointing out that an installation could have happened at the time of the rebuilding of the roof in the 1530s. Before scoffing, it is worth exploring further. The set of arms was not here in 1840. Its placing in the church therefore roughly coincided with Colonel Rushbrooke's reordering. Thus far, simple enough; and yet, the coat of arms is a rather more primitive piece of work than the furnishings of the chapel below. Simply, it looks older. Painted below it on the former rood beam is the motto Dieu et mon Droict.

We know that Henrician coats of arms were put into churches. At the time of the Marian restoration in 1553, they were removed and destroyed. It is simply beyond all theological and political credibility that a set could have survived in situ. No symbol of loyalty to the crown could be used to express disloyalty to that crown. They were all replaced by crucifixes - or, at least, that was the intention.

At Ludham in Norfolk, the tympanum was retained, and the rood painted on it. Elsewhere, roods seem to have been reconstructed enthusiastically, but Mary died before her counter-reformation was solidly in place. Also apparently destroyed, of course, were Edward VI arms, although in practice these must have been few and far between. If most churches obeyed the order to install the royal arms of Henry in the previous reign, we may assume that those which didn't would hardly have been disposed to install them as an act of submission to the lunatic policies of his young son's advisers. There were no Marian arms, and Suffolk's only set of Elizabeth I royal arms is at Preston, a magnificent object.

So, where did this set of arms come from? Is it possible that it could be genuine, and removed from a church by the order of 1553, it survived the 290 years in storage somewhere? And then, Colonel Rushbrooke found it, bought it, and installed it here? Is it even possible that it came originally from this church, and was stored at Rushbrooke Hall?

Or is it possible that the arms are something wholly different, and were never designed for a church? The arms of Henry VIII are also those of Henry VII. Many were produced in the late 15th and early 16th centuries to further the hegemony of the Tudor cause. Rushbrooke was an enthusiastic collector, and might have tracked this set down in a public building, for instance. He was enough of an antiquarian to know that a Henrician set in a church would be unique.

Or perhaps this is all complete speculation, and he made them specially. Cautley hedged his bets, although his posthumous editors scoffed. Mortlock was also tongue-in-cheek about it, but I don't suppose that we will ever really know.

In the years I have been coming here, I have grown very fond of this utterly unusual little church. It is quirky in so many respects, from the aspects already described to the splendid little Flemish roundels of unicorns, which didn't make it into Nowton, the skulls carved deep into the walls of the south aisle, and the great hatchment hanging a few feet above the floor. It was good to see it all again.

Feeling quietly pleased with myself, I closed the door behind me, and set off for Bradfield St George under gathering cloud. Spots of rain greeted me as I reached the Whelnetham road, but I was inside the next church before it began to fall. I opened my bag, and discovered with some dismay that I had left my camera at Rushbrooke. Well, there was nothing else to be done. I got back on and cycled through the rain the couple of miles back up the hill. Of course, it was still there when I got there. But perhaps the ghost of Colonel Rushbrooke had heard me scoffing, and decided that he'd like the last laugh after all.