Other things on this site...

[Blog archives] [Categories]

State of the Map 2019 - solar conversations

I'm just back from State of the Map 2019, the annual global OpenStreetMap meetup. I gave a talk about the solar mapping project (video here) and had lots of great conversations with people.

Here are some notes from conversations I had about solar PV mapping:

  • I had a good talk with the people from Mapillary. Mapillary have a good collection of street-level imagery, they do automatic detection of objects, and they are already integrated into OSM (e.g. a Mapillary JOSM plugin helps people to use their imagery and even their auto-detections). Note that they're a private company and so I guess they don't share their ML methods or pre-trained networks, though they do explicitly share imagery (and some of their detections) with OSM. They do NOT currently have a tag for solar PV or suchlike. Here's a plan:

    1. They're going to add a "solar panel" tag into their online system. This will be about two weeks from now.
    2. People (us?) will then be able to use their online editor to tag solar PV objects in street-level imagery. This does not directly mean anyone's going to be doing any auto-detection. But it does give us a new and different source of imagery to spot solar PV in.
    3. We'll be able to use the Mapilary plugin (I think) to add those streetside-spotted items to the map.
    4. If we can tag a good diverse set of small-scale solar PV - say, a few hundred of them, in different countries - then we will be in a position to start chatting to Mapillary about whether they're interested in training some auto-detection for that tag.
  • In the UK we've mapped solar farms with the power=plant tag, which is good because it makes them clearly different from the individual solar panels tagged as power=generator. However, around the world this isn't the case. Lots of people have mapped solar farms with single large power=generator tags. This is not necessarily terrible because we can identify large objects when we postprocess. It's problematic if people are using single nodes to mark solar farms though, since it's hard to tell the difference between a 10 kW and a 100MW installation without more detail. I hope we can convince the wider community that our way of tagging (which comes directly from the established power tagging) is the way to go.

    • Although the editing tools have a preset for solar panels, they don't have one for solar farms / solar parks. We should try to propose a preset for JOSM and iD. It would fill in the basic tags, and offer a textbox for entering the capacity. Also it'd be good to ofer a text box for the start date, to gather that if known.
    • The default map rendering on openstreetmap.org doesn't render anything for the power=plant tag. This may be acting as an incentive for people to stick with power=generator, because of the visibility of the results. (Some people add landuse=industrial to solar farms, which is no problem for us; I don't know if they're doing it because it renders quite nicely.) We should ask the maintainers of the openstreetmap-carto rendering style if they're willing to render solar farm areas, perhaps just the standard industrial colour.
  • I met Julien Minet who had created a highly detailed service for the city of Brussels. The city commissioned a LiDAR survey of all the buildings, giving them a very detailed 3D model. The developers used this model to predict the solar energy yield for every single roof part (can be multiple on each building!), and made this into a very nice app for people to predict their personal projected benefit of installing solar PV. This is the kind of detail that only a well-organised city can feasibly manage... It's a high-res version of what "open solar map" did for France, which is a more scalable example.

  • How to vectorise blobs from pixelwise detections... it seems it's fairly common knowledge. QGIS at least has a function for it ("polygonize") which offers at least a starting point, though it's very simple-minded (it traces all the edges of pixels - you need to run a "simplify geometries" afterwards). Facebook's new "rapid" ML-assisted editing tool proposes building shapes for the user to add, so they are doing the process too somehow. (I'm pretty confident they're doing that as postprocessing, not some fancy ML that directly spits out geometries.) I don't know which are the current leading methods for this.

  • I like the look of Facebook's approach to machine-augmented editing. They made a modified version of the iD editor (and named it rapiD), which offers machine-proposed areas for the user to confirm-and-edit one by one. I wasn't able to attend their demo though so I don't have more details e.g. whether it's stable, whether we could adapt it.
  • There's lots of latent interest in trying machine learning detectors on aerial imagery - I saw a couple of talks about doing this for other purposes. Largely using methods we'd recognise from other work (i.e. convnets). I saw a talk from IBM using CycleGAN which seemed to me a very odd choice of method, I'd not recommend that! On the other hand, I started to wonder whether we should be organsing a small-scale-solar-PV-detection data challenge. (We don't need to worry about detecting large-scale solar, there's existing work on this already.) We've got sufficient annotations for the UK now to make ML training sets - I guess the key question is whether we can get good enough imagery, and whether it's something that'd be of benefit. We could use aerial or even streetside imagery. As well as Mapillary I chatted to Maxar (aerial imagery providers) - the Maxar fellow mentioned a US company/charity called something like CrowdSpark (I THINK - or maybe it was a different company) that has organised data challenges related to development/aid. If we did put together a data challenge, what approach would be most likely to lead to detections that (a) could help us complete the UK, (b) would help us to scale globally? I don't know if this is worth pursuing though - can perhaps simply rely on existing object detectors rather than assuming it needs novel effort - TBC.

  • I caught up with Jerry (SK53) who was a key player in the OSM UK solar mapping. Didn't manage to catch up with Jez Nicholson - somehow we missed each other!

Overall, there was lots of enthusiasm for the idea of mapping all the solar PV, and I'm really glad to have been at SotM to talk about it.

| openstreetmap |

Solar panels in the UK - 100,000 spotted!

The OpenStreetMap UK community has come together for a 3-month "quarterly project" to find all the solar panels in the UK. And the results so far... wow!

-> We've just reached over 100,000 standalone solar PV installations mapped in the UK!

-> Plus we also have almost 600 solar farms mapped, representing almost 5 gigawatts!

To give you an idea of completeness: the UK government believes there are about 1 million solar panels in the UK, and official solar farm listings have about 8 GW. What we've mapped so far is way more than I thought we'd manage, so I really deeply want to thank every person who contributed, large or small. We can use these data to pilot CO2-saving initiatives - and so can anyone else. Open data for the win!

Here are a couple of simple plots from me, to show where the items are in the UK. (The data I've used are from yesterday, which is why my plot says not-quite-100,000 items.) I've plotted two different types of item: (1) "standalone" solar panels (i.e. ones that are not inside solar farms); (2) solar farms. You'll notice the distributions are really different:

Some of the darkest blobs on the map are evidence of focussed effort. I know that Jerry (SK53) has been mapping around Nottingham, Cornwall and other areas, so may be the source of some of these blobs. We've shown that with some local effort, and a bit of scanning through imagery, a lot of this can be done.

You'll notice that many of the solar farms are in the southern part of the UK, at least according to our data. That's not unexpected!

For more detailed data breakdowns, you can peruse Gregory Williams' solar mapping analysis site.

We have more time to go until the end of the quarter... and we'll be able to use the data for sustainable energy projects whenever you contribute - so please do join in!

| openstreetmap |

Tool needed: simple solar spotter

The OSM UK community is great, but it's hard to guarantee we can do the detective work to spot all 800,000 solar PV installations. There's a "long tail" of solar panels tucked away down side-streets. It's very much a needle in a haystack, and we would benefit from as many hints as possible.

(We're already working with (a) machine learning and (b) official data sources. They're good sources of hints too, but not the full picture.)

We could ask the general public for help with this. Almost everyone must pass a solar panel during their daily commute, their weekend stroll, or suchlike. But we can't expect the general public to use map editing tools, or in fact anything that requires technical commitment or expertise. Also nothing that requires login or user registration.

Can we make a tool that makes it so simple, that many thousands of people can send in just one or two sightings each?

We don't have the resources to make a fancy phone app. (And would people use it if we did?)

Option one: "drop a pin in a map" approach. Provide a simple webpage which lets people, with no login required, put a pin at a location where they think there's a solar panel. This is fairly easy to code (and could use the OSM Notes API, e.g. with a specific pre-agreed template for the Note text). However it's not ideal for people out-and-about with a smartphone, since it's all about the top-down birds-eye view.

Option two: people can take a photo of a solar panel they see, and post it to a service they already use. (Smartphones often record GPS location along with photos.) Posting to Twitter (with a particular hashtag) would be easy to set up and to scrape. However not everyone uses Twitter. Loads of people use WhatsApp. Can they report their solar spottings directly through a WhatsApp number? It would need someone to set up a number, fine - and then the coding required is to create a system that can slurp whatever photos were sent in, do a bit of sanity-checking and maybe some basic "bot" interactivity, and output a dataset of suggested-geolocations for panels. (These suggested-geolocations would not go into OSM directly, they're not appropriate for that. We can simply provide them as a dataset for mappers to refer to.)

Thanks to Max+Esther at 10:10 for suggesting the second idea. I approached them because 10:10 has previously been involved in something a little bit similar (a mobile app for spotting rooves that would be good places for new panels). They've clearly got the right idea for the kind of simple everyday interaction that's needed.

| openstreetmap |

Where are the solar panels in Britain?

Where are all the solar panels in Britain? Are they in the south? The sunny east? The countryside, the city?

The UK's office "Ofgem" publishes open data about the solar PV installations that they know about. In the latest "feed-in tariff" (FiT) data, there are about 800,000 of them. The "installed capacity" adds up to about 4.9 gigawatts, about half of which comes from big industrial field-scale installations and half from domestic rooftop solar.

It would be handy to know where the solar panels are - for example, if you're searching for solar panels to map...

For privacy purposes, Ofgem don't publish exact locations, nor unique IDs, in their big spreadsheet. So the data aren't perfect for mapping, but they do give us the postcode district for 90% of these 800 thousand. So, using that postcode info, I've taken their data and simply plotted them on a choropleth. Let's take a look!

Before you look, please note that I'm plotting the raw numbers per postcode district, and NOT normalising the data to account for the size of the district. This partly explains why the plots look "dark" in the regions (such as London) which are chopped up into lots of small districts. Smaller districts should have fewer things in... but on the other hand, smaller districts are supposed to equate to higher density of households, so maybe the postcode district is a good unit of analysis after all.

Here are the plots - three plots showing, respectively, the raw number of installations per district, the total installed capacity in each district, and finally to get an idea of household density I also plot the number of households there are in each district according to census data:

And here's a CSV spreadsheet of the summary FiT numbers I used to plot these. Sorry for not showing (Northern) Ireland, it's not in the data I found.

(The CSV and the images are all derived from Ofgem's FiT data which are published under the Open Government Licence.)

Note that there are A LOT of caveats about this data. About 10% of the solar installations (80 thousand!) whose postcode district was listed as "unknown". Also some postcodes are allegedly not quite right (e.g. some of them are the postcode of the person who registered, not the location of the thing itself). Some of the installations they've listed might have been discontinued, and we don't really have much way of knowing. Oh, and... the postcode area data I'm using seems to have some omissions, hence the occasional white gap in Britain. But notwithstanding all that, this gives us some indication of the distribution.

One thing that pops out to me is that these three plots don't seem very correlated. I'd have expected them all to be highly correlated. For some reason it looks like a relatively high number of small-capacity installations across Yorkshire down into Essex. There's plenty of regional variation and clustering, which may be due to geographical/weather differences, or perhaps to local initiatives.

| openstreetmap |

Spotting solar panels in London

Jack had this great idea to find the locations of solar panels and add them to OpenStreetMap. (Why's that useful? He can explain: Solar PV is the single biggest source of uncertainty in the National Grid's forecasts.)

I think we can do this :) The OpenStreetMap community have done lots of similar things, such as the humanitarian mapping work we do, collaboratively adding buildings and roads for unmapped developing countries. Also, some people in France seem to have done a great job of mapping their power network (info here in French). But how easy or fast would it be for us to manually search the globe for solar panels?

(You might be thinking "automate it!" Yes, sure - I work with machine learning in my day job - but it's a difficult task even for machine learning to get to the high accuracy needed. 99% accurate is not accurate enough, because that equates to a massive number of errors at global scale, and no-one's even claiming 99% accuracy yet for tasks like this. For the time being we definitely need manual mapping or at least manual verification.)

(Oh, or you might be thinking "surely someone officially has this data already?" Well you'd be surprised - some of it is held privately in one database or other, but not with substantial coverage, and certainly almost none of it has good geolocation coordinates, which you need if you're going to predict which hours the sun shines on each panel. Even official planning application data can be out by kilometres, sometimes.)

Solar panel aerial image examples

Jerry (also known as "SK53" on OSM) has had a look into it in Nottingham - he mapped a few hundred (!) solar panels already. He's written a great blog article about it.

This weekend here in London a couple of us thought we'd have a little dabble at it ourselves. We assumed that the aerial imagery must be at least as good as in Nottingham (because that's what London people think about everything ;) so we had a quick skim to look. Now, the main imagery used in OSM is provided by Bing, and unfortunately our area doesn't look anywhere near as crisp as in Nottingham.

We also went out and about (not systematically) and noticed some solar panels here and there, so we've a bit of ground truth to put alongside the aerial imagery. Here I'm going to show a handful of examples, using the standard aerial imagery. The main purpose is to get an idea of the trickiness of the task, especially with the idea of mapping purely from aerial imagery.

It took quite a lot of searching in aerial imagery to find any hits. Within about 30 minutes we'd managed to find three. Often we were unsure, because the distinction between solar panels, rooftop windows or other rooftop infrastructure is hard to spot unless you've got crystal-clear imagery. We swapped back and forth with various imagery sources, but none of the ones we had available by default gave us much boost.

While walking around town we saw a couple more. In the following image (of this location), the building "A" had some stood-up solar panels we saw from the ground; it also looks like "B" had some roof-mounted panels too, but we didn't spot them from the ground, because they don't stick up much.

Solar example

Finally this picture quite neatly puts 3 different examples right next to each other in one location. At first we saw a few solar panels mounted flush on someone's sloping roof ("A"), and you can see those on the aerial - though my certainty comes from having seen them in real life! Then next to it we saw some stood-up solar panels on a newbuild block at "B", though you can't actually see it in the imagery because the newbuild is too new for all the aerials we had access to. Then next to that at "C" there definitely looks to be some solar there in the aerials, though we couldn't see that from the ground.

Solar example

Our tentative learnings so far:

  • We will need to use a combination of aerial mapping and on-the-ground "solar spotting" from people.
  • Whether on-the-ground or aerial, it's often hard to get a clear idea of the size of the installation. In lieu of that, maybe it's fine to map them as points rather than areas. People can come along later and tell us the actual power ratings, efficiencies etc.
  • We will need to make the most of heuristics about where to find solar panels. For example Jerry notes that social housing is relatively likely to install solar, and I've noticed it on plenty of schools too; there may also be vendor/official lists (e.g. planning applications) out there - we'll need multiple sources to get a well-rounded coverage.

See Jerry's blog for more learnings.

There are plenty of virtuous feedback loops in here: the more we do as a community, the better we'll get (both humans and machines) at finding the solar panels and spotting the gaps in our data.

| openstreetmap |

In mainland Britain, you are never more than 34 miles from a pub.

In mainland Britain, you are never more than 34 miles from a pub.

This and other geo-factoids available from my new web service. (I've named it "Feet From A Rat" in tribute to this hoary old urban legend.)

| openstreetmap |

OpenStreetMap UK: what should we do this year?

As a contributor to OpenStreetMap, one thing I've been wondering recently is what sort of map data should we collect for the UK, now that the coverage has already got good. Since OpenStreetMap generally has great coverage of the UK, when you're out and about with a printed-out map and …

| openstreetmap |

Rejigging the OpenStreetMap browse page

On OpenStreetMap, I find the /browse/ pages really useful for getting a quick summary of an "object" in the map. It shows when it was edited, shows all the tags, etc.

However, I have two issues with it:

  • The use of space isn't ideal. There's plenty of unused space which …
| openstreetmap |

Diversity and OpenStreetMap

The big annual meetup of OpenStreetMap folks was last week and it was full of interesting talks. The diversity of people seemed pretty good relative to a lot of the meetups I end up at (open-source software, experimental music, computer science, you know, that kind of thing), but still, the …

| openstreetmap |

OpenStreetMap: animated dataviz of edits per year

Another iteration of my visualisation of OpenStreetMap edits - here's an animation showing, for each year 2005-2012, the density of edits according to their geographic location:

Animation showing the density of OSM edits in 2005-2012, divided by population density.
Animation showing the density of OSM edits in 2005-2012, divided by population density.

The upper plot is the raw edit density. The lower one (which I think is more illuminating) is the edit density per unit population, as …

| openstreetmap |