When we first released ImproveOSM, the Telenav initiative to share millions of potential missing road, turn restriction and one-way locations based on massive anonymized GPS signals from drivers, we also launched a custom editor based on iD.
The editor provided an easy way to inspect ImproveOSM data and make OSM edits based on it.
Since then, we have worked hard to build out the functionality of the ImproveOSM JOSM plugin, and worked with the iD team to integrate ImproveOSM into the main OSM web editor.
These developments have made the custom editor obsolete, and we have decided to take it offline.
If you are interested in using ImproveOSM data to edit OpenStreetMap, please check out our JOSM plugin or select the ImproveOSM overlay in iD. You can also download the raw data from our ImproveOSM web site.
Thanks for using ImproveOSM and making OpenStreetMap better!
Last December and January, OpenStreetCam held a image collection competition in Australia and in New Zealand. The three mappers in each country who collected the most points during the months of December 2018 and January 2019 could each win a gift card: $100 for the winner, and $25 for the second and third place. We just announced the winners to the communities in both countries. Congratulations to steve91, robbie-bloggs, ConsEbt, david-blyth, ivss-xx, and nicknz!
We decided to do these competitions because we wanted more mappers in Australia and New Zealand to get acquainted with OpenStreetCam, and consider contributing to this free and open platform for street-level images. The more contributions, the more help OpenStreetCam can be for OSM mappers! There weren’t many contributions in either country yet, and if you go to the OpenStreetCam web site, you’ll quickly see that there are still large gaps to fill. Still, OpenStreetCam coverage grew by 800% since the beginning of December.
Head over to the ImproveOSM Blog for a step-by-step guide on how to get started with OpenStreetCam yourself!
OSM mappers can use the OpenStreetCam images to help with mapping. You can’t see everything from an aerial image. Signs are a great example of useful mapping information that requires an on the ground perspective. This is where OpenStreetCam is particularly handy, because we detect an increasing diversity of signs that appear on the photos for you automatically, using an open source machine learning platform.
For the sign detection platform to work and detect a variety of signs reliably, it needs training data. Your contributions during this competition have been invaluable to reach that goal. Our Map Team looked at tens of thousands of images collected by the community during the competition in Australia and New Zealand, and validated more than 160,000 traffic signs found in these images. After feeding that data into the platform, we can now reliably detect more than 80 types of signs in Australia and New Zealand. As we continue to look at more images that you contribute, the system will get smarter and we will detect more different types of signs.
Do you want to help train the OpenStreetCam sign recognition AI? You can do this right from the OpenStreetCam web site. Read all about it in this blog post.
In this post, we would like to guide you in making your very first contribution to OpenStreetCam.
In this post, we would like to guide you in making your very first contribution to OpenStreetCam. There are already 190 million images on OpenStreetCam covering more than 5 million kilometers of road, so obviously getting started is easy enough that it can be done without much guidance 😁 But in case you do need a little encouragement to record your first trip, or just want to see how it works before you try it yourself, read on.
The first thing you want to do is download the free app. OpenStreetCam apps exist for Android and iOS.
When you first run the app, it will give a quick introduction about OpenStreetCam. Flick through that to get to the main screen. Then go to your Profile, where you can log in.
You can create an OpenStreetCam account by logging in with your existing OpenStreetMap, Facebook or Google accounts. You won’t need to create a separate password.
After logging in through the platform of your choice, you will see your new OpenStreetCam profile 🙂
Your profile will look a little empty compared to mine, but we are here to change that! Let’s go out and drive some.
You will need some sort of phone mount in your car so you can point the camera straight ahead with a clear and unobstructed view of the road ahead. I use an iOttie brand mount (an older version of this one) but any mount that will hold your phone in landscape mode reliably will do.
You will also want to connect your phone to power. We have spent a lot of time optimizing the app, but the recording still drains the battery quite fast.
Okay, we’re almost ready to go. We just need to start recording mode so the app will start taking pictures as you start moving. Before you do that though, take a moment to scroll around the map looking for streets that have no purple lines. That means that nobody has captured any images there yet, so those streets are extra valuable. (You get 10x points for them too.)
When you’re done with that, press the blue camera button to start recording mode.
You may notice that the app mentions a thing called ‘OBD’. This refers to a port in your car that transmits data about the current state of the vehicle. Using a compatible OBD dongle, OpenStreetCam can use this for improved location accuracy. This is optional but gives you twice the points if detected! If you want to learn more, drop us a line.
The app will not immediately start taking pictures. Using your phone’s built in sensors, it will detect when you start and stop moving. As long as you’re stopped, no pictures are taken. This saves space, time spent uploading, and mappers wading through duplicate images of the same location.
In recording mode, you can switch between a big camera view and a small minimap, or the other way around. You switch by tapping on the minimap / mini camera in the left bottom.
As you drive, you will see your points increase as well as some other basic trip stats like number of pictures taken, space used, kilometers driven and recording time. As I mentioned before, roads that nobody has captured before are worth 10x the points. As you collect more points, you get higher up in the leaderboards and level up!
When you’re done driving, hit the record button to end the trip. You will now see a summary screen for your trip, showing where, how long and how far you’ve driven, as well as how many points you have collected on this trip.
Now that you’re done collecting your first images, it’s time to upload them to OpenStreetCam. This does not happen automatically by default (but you can go into settings to change that.) So as soon as you’re connected to wi-fi, go back to the app and go to ‘Upload’. There you will see the trip you just created.
You can tap on the trip to get more details. One cool thing you can do is ‘scrub’ through the trip.
Tap ‘Upload all’ in the top right corner to upload. You will notice that the file size is actually relatively small. That is because internally, the app compresses the images into a video stream that is unpacked into separate photos again at the server side, saving you time and upload bandwidth.
Once the upload is finished, you can go to openstreetcam.org and log in there. Use the same login method you used in the app, so if you used OpenStreetMap to log in on the app, use OpenStreetMap as your login provider on the web site as well.
Once you’re logged in, you can go to your profile to see your trip.
You can click on the trip to see the uploaded images. (Here is the trip I recorded for this demonstration.)
Notice that I should have wiped the snow ❄ off my car before I started recording.. 😬
Finally, I highlighted two icons you see on the lft hand side of your trip detail window. If you are in the U.S., you may see a number badges. They indicate how many street signs were recognized (the bottom one) and how many of those represent data that doesn’t seem to exist in OSM yet. That’s for a future post though!
Telenav open-sourced the machine learning based sign detection platform that powers the automatic detection of nearly 100 sign types in the OpenStreetCam images you contributed. You can already see these detections in the latest version of the OpenStreetCam JOSM plugin to help you map, and iD integration will come soon as well.
Machine learning gets better with training. The more known instances of a particular sign that are fed into the system, the more reliable the automatic detections for that sign type will become.
Our Map Team has spent thousands of hours manually tagging and validating traffic signs in images, and the resulting training data is open source as well. But did you know you can help improve the detection system yourself as well? Let us show you how.
If you go to the trip details on the OpenStreetCam web site, you will see three ‘tabs’ on the left. The first one takes you to the main trip info. The second one takes you to an OSM edit mode, that lets you quickly go over detections and see if they need to be added to OSM. (Separate post! The third tab is the sign validation mode. If the tab icon has a number with it, there are unverified signs to work on.
The bottom part of the screen shows all detected signs. The ones that have been validated already will have a green checkmark with them. The ones that have been invalidated will have a red ‘X’.
You can validate or invalidate the automatic detection if the sign on the image exactly matches / doesn’t match the automatic detection, by clicking the corresponding button on the left.
Power Validator Workflow
You can validate entire trips with many detected signs very quickly by using some of the power functions available:
Next to the trip slider, underneath the image, you will find a small magnifying glass button. Clicking this will automatically zoom and pan the image to the detection
Use Cmd (Mac) / Alt (Windows / Linux) and the left and right arrows to quickly jump to the next detection
Use Cmd / Alt up and down to validate or invalidate the currently highlighted detection.
It has been an exciting summer! Besides our regular work, there was the annual State of the Map conference that we were all really looking forward to. We launched a new ImproveOSM web site. OpenStreetCam dash-cams are distributed to OSM US members. And more. Read all about it in our Summer Dispatch below!
State of the Map
Quite a few of us got to go to State of the Map in Milan, Italy! Our team hosted four presentations at the conference, and we are really happy with the interest and feedback we received. We made a lot of new map friends as well!
All SOTM presentations were recorded and posted on YouTube, so if you missed any of us, you can watch the presentations at your leisure:
Alina and Bogdan talked about our ongoing effort to extract signs and other meaningful data from the more than 160 million OpenStreetCam images.
We also had a booth at the conference where we talked about ImproveOSM and OpenStreetCam, and where 6 lucky winners received a Waylens OpenStreetCam dashboard camera!
We continue to map in Canada, the United States, and Mexico. As always you can track our work on GitHub. We have been focusing a lot on adding missing road names for the larger metropolitan areas in the US. Our typical workflow is to identify local government road centerline data sources, verify the license, process them with Cygnus to find changed / new names, and manually add the names if we can verify them.
Right on time for State of the Map, we launched a complete redesign of improveosm.org, our portal for everything Telenav❤️OSM. The new site gives you quick access to our OSM initiatives, data and tools. Check it out!We also released more than 20 thousand new missing roads locations. These are added to the existing database of currently more than 2.4 million missing road locations. An easy way to start editing based on these locations is to download the ImproveOSM plugin for JOSM.
The steady growth of OpenStreetCam continues. Almost 4.5 million kilometers of trips are in the OSC database. This amounts to about 165 million images!
We started a collaboration with OpenStreetMap US to run a Camera Lending program. Through the program, OSM US members can apply to borrow a custom Waylens Horizon camera for up to three months. The camera captures high resolution images for OSC and uploads them automatically. Almost 20 mappers have a camera already, and they have driven about 30 thousand kilometers in the past couple of months!
That’s a wrap for our summer dispatch folks! Thanks for reading and keep an eye on the blog for more from the Telenav Map Team. Be sure to follow us on Twitter as well @improveOSM and @openstreetcam. 👋🏼
The Telenav OSM team just released a new version of the OpenStreetCam JOSM plugin. The major new feature is the ability to show and manipulate street sign detections. Images in only a few areas are currently processed for sign detection, so it’s not very likely that you will see anything yet, but that will change over time as we catch up processing over 140 million images.
To enable detections, right-click on the OpenStreetCam layer in the Layers panel, and check ‘Detections’ under ‘Data to display’. You can filter the detections by the following criteria:
Not older than — show only detections (or images) from that date or newer.
Only mine — show only detections / images from my own OSM / OSC account.
OSM Comparison — show detections based on comparison with OSM data:
Same data — Only show signs that have corresponding tags / data already mapped in OSM
New data — Only show signs that do not have corresponding data in OSM and need to be mapped
Changed data — Only show signs that have existing tags in OSM but the value is different (for example a 50 km/h sign and the OSM way is mapped as 60 km/h)
Unknown — No match could be made between the detected sign and OSM data
Edit status — show detections based on manually set status of the detection:
Open — new detection, status not changed yet
Mapped — manually marked as mapped
Bad sign — manually marked as a bad detection
Other — other status
Detection type — show only signs of the selected types.
Mode — Show only automatic detections, manually tagged detections, or both.
For the filters OSM Comparison, Edit status and Detection type, you can select multiple values by using shift-click and command/ctrl-click.
In the main editor window, you can select a sign to load the corresponding photo, which will show an outline of the detected sign. If there are multiple signs in an image, you can select the next one by clicking on the location again. (This is something we hope to improve.)
In the new ‘OpenStreetMap detections’ panel, you can see metadata for the detection, and set the status to Mapped, Bad Detection, or Other. By marking signs that are not detected correctly as Bad Detection, you hide them from other mappers, and we will use that information to improve the detection system.
The plugin is available from the JOSM plugin list, and the source is on Github.
Our ImproveOSM pipeline produces a pretty impressive number of suggested roads missing from OSM, missing oneway tags, and missing turn restrictions, based on analysis of billions of GPS data points. We make the results available as frequent data dumps in CSV format. In this post, I want to look at a way to integrate this data into your OSM mapping workflow.
If you just want to see ImproveOSM data in JOSM wherever you are currently mapping, you can just use the ImproveOSM JOSM plugin. For advanced users who want more flexibility, or who want to use this data in different ways, this post offers some guidance.
The data dumps are available from here. For this example, I will work with the most recent Direction of Flow data file. This highlights ways with potential missing oneway tag. After downloading and unzipping it, you will have a CSV file of about 16.5 megabytes that looks like this:
Since the theGeom field is in WKT, you can import it as a layer in QGIS pretty easily. Let’s fire up QGIS (I use 2.18) and add a Delimited Text layer.
In the dialog, select the downloaded CSV file as the file source. Set the delimiter to semicolon. QGIS detected for me that the geometry was in the theGeom field, and of type WKT, but you can set that manually if needed:
Upon clicking OK, QGIS wants us to define which CRS the coordinates are defined in. Select WGS84.
Now, we have a layer of line geometries that correspond to OSM ways that may be missing a oneway tag.
To make the file more manageable, let’s limit our selection to one country. I get country boundaries from Natural Earth (a fantastic resource!). After adding the country borders to QGIS, I can perform a spatial query. Before you do this, select the country you are interested in. I pick Mexico as an example.
Bring up the Spatial Query window. If you don’t see this menu item, you will need to enable the Spatial Query plugin.
Select the ImproveOSM layer as the source, and the Natural Earth layer as the query layer. Make sure to check the ‘1 Selected geometries’ checkbox, so we limit our query to Mexico.
The matching features will now be selected in the ImproveOSM layer. Make sure that layer is selected in the Layers Panel before you select Layer -> Save As.. from the QGIS menu. In this dialog, choose GeoJSON as the output type. Select a destination filename. Make sure that the CRS is set to WGS84. Make sure the ‘Save only selected features’ is checked, and Save.
Now you have a GeoJSON file with all OSM way geometries that may need a oneway tag. You can load this file into JOSM, using its GeoJSON plugin. To organize your work going through these, I would recommend using the Todo plugin and add the GeoJSON features to the todo list.
GPS technology is growing fast along with the mapping industry. There are a lot of navigation apps on the market, trying to step forward using different technologies and map data. As you know, behind all its routing processes and algorithms, a good routing application must have a good quality map.
Our team, Telenav, is developing an Open-Street-Map (OSM) based app, called Scout. Why Open-Street-Map? In my opinion OSM is a good choice because it’s an open, fast growing and well built-up community map. With its great involvement and commitment, the community is constantly trying to keep all the map features from OSM up to date.
For the purpose mentioned above, Telenav decided to improve the quality of the OSM map features, used for navigation, in Phoenix, Arizona.
From September until now, the whole project workflow was realized in 3 main steps:
Step #1. Research and get in touch with local community (research on local open source data, keep in touch with local OSM users, official traffic signs and driving rules, research on HOV and toll roads, roads under construction, other specific road features of Arizona State).
Step #2. Build-up process:
Road geometry, Road Name, One Way, Gates and Other Geometric Feature (turning circles, turning loops, etc.)
Turn Restrictions and Traffic Lights
Lane and Turn Lane Info, including HOV Lanes and Toll Roads
Speed Limit Editing
Step #3. Quality Assurance (QA for every feature edited by our team in OSM, some road name special issues, repairing broken relations, final edits and corrections based on Improve OSM plug-in, Osmose and Keep Right errors, tile-by-tile verification).
The first step was a very important one because the team had to get in touch with the local community and mapping rules. Also, we researched other open source and free of charge data beside the already well-known sources used by OSM users (TIGER Roads, Bing, Digital Globe, Open-Street-Cam, Mapillary). The under-construction roads found in this step were permanently monitored, to be edited later.
The second step was the most time consuming and it included the editing and reviewing of all the most important navigable OSM features, starting from motorway, trunk, primary, secondary, tertiary, residential and service ways. The whole editing process lasted for about 3 moths and the workflow was mainly based on a tile-by-tile method. We also edited based on way category or route reference. During this period, we helped the community to increase the OSM quality as you can see below:
The third step was the last but not the least important. Because we wanted to have good quality features in OSM, we had to make a closure check on our edits, already made in the build-up process and where it was necessary, to fill the gaps. So, we used different QA procedures, queries, other plug-ins, error identifiers and even tile by tile verification.
For the queries based QA, we ran some predefined scripts based on OSM datasets, using pgAdmin and PostgreSQL. The queries were aimed mainly at finding:
Wrong one-way direction
Wrong number of lanes
Long relation members
Turn Restrictions with unusual number of members
Ramp has name or reference number
Similar names and destinations
Detect possible roundabouts, etc.
The main JOSM plug-in used to complete the missing roads was Telenav ImproveOSM (also available on https://improveosm.org). The plug-in focuses especially on missing roads, one ways and turn restrictions.
Doing the QA, we had an important overview of all our edits, especially of our wrong ones. In this step, we resolved also a name tag issue we’d came across during step #2. Step #3 represents confidence of a good quality editing process for the Telenav team.
When we needed a double check of our work, the community was a good help, giving us important feedback. For the next months, we’ll surely keep in touch continuously with the local OSM users, looking forward for their feedback to keep up a good maintenance work. The top editing users were the most helpful users, giving us precious and pertinent feedbacks.
In the GIF below you can see an evolution of Telenav mapping team edits during the last moths, in Phoenix, Arizona. Darker colors indicate high density and brighter colors low density.
Looking from a map analyst point of view, turn restrictions are some of the most important features a map can have. Turn restrictions influence a lot the way a route is made. If they are wrongly edited, they can cause bad routing, having big consequences on travel time, travel directions, maneuvers and so on. That’s why we decided to talk a little bit more about this case: editing turn restrictions.
We worked on this issue for 3 months, starting from November 2017. We succeeded to review a large amount of intersections between motorway, trunk, primary, secondary, tertiary, residential and service ways from Phoenix area.
During our project we accomplished:
to add new turn restrictions
to correct some of the damaged ones
to remove some of the wrong ones.
The main sources of adding or editing turn restrictions were open-source:
Satellite imagery: Digital Globe, Bing, Esri World Imagery
Street level imagery: Open-Street-Cam (OSC), Mapillary.
The software used in editing OSM was JOSM, an extensible editor for Open-Street-Map for Java 8. Also for a better visualization we used different Map Paint Styles (MapCSS) and thematic Layers (Open-Street-Cam, Mapillary, Mapillary object layer).
With a self-developed procedure, we identified all that could be a turn restriction sign in all the OSC track photos, overlapping our area. In the first part of our project we looked over 1723 turn restriction signs identified in Open-Street-Cam, spatially distributed as in map below. Before adding any turn restriction, every detection or sign found in Open-Street-Cam was first validated based on the open sources first mentioned.
In the second part, we managed to identify turn restriction signs also based on Mapillary Imagery and Mapillary Object Layer.
All the reviewed turn restriction signs summed in the end 3289, from which 350 turned to be missing from Open-Street-Map. During this task we also reviewed 11932 and added 376 new traffic lights.
The final step for our task was to do some QA testing. Firstly, we used a query based QA to verify the quality of all turn restrictions from OSM Phoenix area. For this, we ran some predefined scripts based on OSM datasets, using pgAdmin.
The queries aimed mainly:
Long relation members
Turn Restrictions with unusual number of members
“Odd” tagging in existing turn restrictions
Conditional turn restrictions with old tagging scheme
Example of query used to identify unusual turn restrictions from OSM:
We wanted to make sure we’re covered all turn restrictions. So, we used the Telenav ImproveOSM plug-in in JOSM. The plug-in highlights the possible turn restrictions locations based on road geometries and other probe data.
These kind of errors identified by us during the query based QA can also be found on http://osmose.openstreetmap.fr/en/map/ and https://keepright.at/ sites. These two sites contain datasets with all kinds of errors from OSM. The datasets were downloaded, extracted using some Python scripts, clipped after our bounding box and filtered using some SQL queries in pgAdmin. The results were exported and compared with the initial errors resulted in the first QA step and where it was necessary, the turn restrictions were modified or corrected.
The heat map below presents all the turn restrictions edited by the Telenav team, from November until now, in Phoenix. Darker colors indicate high density and brighter colors low density.
Cygnus is the Telenav Mapping conflation tool. We use it a lot internally to compare approved external data sources with existing OSM data, but there is also a public version. We outlined how it works in an earlier blog post. In this post, I want to highlight some of the newer features in Cygnus. These new features are based on the feedback from our team of Map Analysts, who use the tool in their day to day work.
Discarding Very Short Segments
Cygnus outputs the differences in geometry between existing OSM data and the spatial data that we want to use to improve OSM. Sometimes, when the differences are very tiny, Cygnus used to export very short ways. These are not really meaningful enhancements, and clutter up the result data. Therefore, we implemented a length filter. Ways shorter than a defined length threshold will not be included in the output. Based on experience, we set the default to 5 meters. In the internal (command line) version our team uses, this can be tweaked using a parameter. In the public web version, this is not yet possible. We can consider adding it if there is sufficient demand.
When comparing road geometry, Cygnus not only compares geometry, but also road names. An annoying side effect we noticed is that road names are often not exactly the same in OSM as they are in the external data we compare with. This does not mean that the external data is necessarily better. For example, OSM could say that the name of a road is “River Road”, and the external data source could say it is “River Rd”. This is not a meaningful difference, and we would want to exclude those in most cases. So we added a string distance based threshold in Cygnus to filter out similar strings. It is set to a sensible default which, again, can be tweaked in the command line version we use internally, but not yet in the web version.
Another Cygnus improvement related to road names is to ignore name differences on certain types of ways: roundabouts and service roads. Roundabout ways in OSM do not have names by convention, unless the roundabout itself has a name, so they should generally not be added. Service roads technically can have names in OSM, but it is not common. In external data, they do sometimes have names, but if they do, it usually does not make sense to add them to OSM. Based on our experience, they often have descriptive names like ‘driveway’ or ‘access road’ in the source data.
You can use Cygnus yourself by going to http://cygnus.improve-osm.org/ and uploading your source data file. You need to do a fair amount of work to prepare the source data: translating the source attributes into valid OSM tags, and converting to OSM PBF. And always remember to consider carefully what you do with the result. Cygnus is not designed to be an automated import tool. Every suggested change should be manually reviewed.
Let us know how you have used, or would like to use Cygnus!