For our holiday we found a very promising cottage just over the Forth Bridge from Edinburgh, called "Woodland Cottages" on their website. (For address purposes, the cottage is Dales Farm Cottages - here's a link to the location on Google Maps.)
The cottage itself was really good. One of the nice touches is that the owners make wood furniture, which means that (as well as allowing us access to the small wood out the back) various pieces of furniture and fittings throughout the cottage were individually-crafted wooden items, giving the cottage a sense of character. The welcome from the owners was also very good - aside from the mere fact of letting us in and showing us round, they provided a bowl of fresh fruit and a bottle of wine to get us started, as well as various essentials like teabags and porridge oats.
There was a coal/wood fire which was nice to use, although the weather was quite warm so it wasn't often needed. One thing to note is that the cottage sits on quite a quite well-used B-road, and at the back of the woods is a motorway, so if you're looking for an absolutely tranquil location, this isn't it. But it's very near Inverkeithing, a nice small town to pop into for a pub for the evening or for a fish or haggis supper. And the trains from Inverkeithing to Edinburgh are convenient enough that we used them a lot.
Inverkeithing is the most pleasant small town in the area, judging by our experience, and one of the oldest too. More recently-built towns like Dalgety Bay and Kelty are kind of soulless and don't have much to offer. The walk around Loch Ore is very nice but there's not much point pressing on into Lochore town unless you want to visit the Co-op. But we did have some nice walks and we were blessed with the weather. One of the nicest walks was (after getting the train into Edinburgh) the walk up Arthur's Seat. It's so odd that such a goodly chunk of hill/crag is right in the middle of a capital city. Edinburghians (?) seem to appreciate it too.