A fresh Cygnus+ version

Cygnus is a tool that helps the community improve the OSM map by comparing it with an external (local) map, and detecting differences between the two maps. The differences detected include road geometry and certain tags of interest (e.g.: ‘name’, ‘oneway’, ‘ref’, ‘maxspeed’) that are missing or have different value from OSM.

In this latest version of Cygnus, we have introduced new features and made a few overall improvements.

New Features

Minimum Way Length Distance

A new setting was added in this latest version: the minimum distance by which a way’s length in the OSM map may be extended using the external map.

Conflation setting which is defaulted to 15%. Maximum can be 50%

Extending a way’s length can be very useful where section of a road may be missing or a new road needs to be added. Existing ways will be extended when the local map has a significantly longer way (as defined by the maximum connect distance). If it is possible to connect the extended way to any new ways, this will be done automatically.

New Grouped Settings and Checkboxes

We have grouped settings for ease of use into conflation, way-related and tag-related groups.

New checkboxes were also created for adding/extending ways (to allow more refined usage), adding/changing tags and specifying a minimum lexical difference.

Conflation Results

The conflation zip file now contains a narration log detailing each change that is proposed by the conflation archive. The Cygnus+ output explains in the ‘telenav:action’ tag exactly what action was taken on each way (this same output goes into the narration log).

Improvements

The conflation algorithm was improved and also the readability of parameters on jobs in the queue.

We hope you benefit from those changes, and let us know if you have any questions or ideas for further improvements.

Summer Dispatch From The Telenav Map Team

It has been an exciting summer! Besides our regular work, there was the annual State of the Map conference that we were all really looking forward to. We launched a new ImproveOSM web site. OpenStreetCam dash-cams are distributed to OSM US members. And more. Read all about it in our Summer Dispatch below!

State of the Map

Quite a few of us got to go to State of the Map in Milan, Italy! Our team hosted four presentations at the conference, and we are really happy with the interest and feedback we received. We made a lot of new map friends as well!

All SOTM presentations were recorded and posted on YouTube, so if you missed any of us, you can watch the presentations at your leisure:

Alina and Bogdan talked about our ongoing effort to extract signs and other meaningful data from the more than 160 million OpenStreetCam images.
Bogdan and Armin talked about how we verify our edits using a custom, open source workflow.
Adriana talked about our brand new metrics dashboard giving deep insights into the trends in OSM data.
Istvan talked about custom JOSM styles and scripts he uses to make his mapping life (and yours!) easier.

Alina and Bogdan presenting our Machine Learning stack at SOTM 2018

We also had a booth at the conference where we talked about ImproveOSM and OpenStreetCam, and where 6 lucky winners received a Waylens OpenStreetCam dashboard camera!

Excited crowd right before one of the Waylens cameras is being given away!

Mapping

We continue to map in Canada, the United States, and Mexico. As always you can track our work on GitHub. We have been focusing a lot on adding missing road names for the larger metropolitan areas in the US. Our typical workflow is to identify local government road centerline data sources, verify the license, process them with Cygnus to find changed / new names, and manually add the names if we can verify them.

Local road centerline data the team identified in Colorado

We are excited that the US community is looking to build an overview of available road centerline databases from (local) governments. We hope the ones we identified can help bootstrap this initiative.

We also published some MapRoulette challenges around this topic.

ImproveOSM

Right on time for State of the Map, we launched a complete redesign of improveosm.org, our portal for everything Telenav❤️OSM. The new site gives you quick access to our OSM initiatives, data and tools. Check it out!We also released more than 20 thousand new missing roads locations. These are added to the existing database of currently more than 2.4 million missing road locations. An easy way to start editing based on these locations is to download the ImproveOSM plugin for JOSM.

Locations of the new Missing Roads locations

OpenStreetCam

The steady growth of OpenStreetCam continues. Almost 4.5 million kilometers of trips are in the OSC database. This amounts to about 165 million images!

We started a collaboration with OpenStreetMap US to run a Camera Lending program. Through the program, OSM US members can apply to borrow a custom Waylens Horizon camera for up to three months. The camera captures high resolution images for OSC and uploads them automatically. Almost 20 mappers have a camera already, and they have driven about 30 thousand kilometers in the past couple of months!

The passenger’s seat of our Camera Man ToeBee, as he gets ready to dispatch a bunch of Waylens cameras

That’s a wrap for our summer dispatch folks! Thanks for reading and keep an eye on the blog for more from the Telenav Map Team. Be sure to follow us on Twitter as well @improveOSM and @openstreetcam. 👋🏼

Map Metrics for OSM are now available

Telenav’s OSM team just released a portal where you can view different metrics on OSM.

Unlike other metrics views that are already available, this new tool for the OSM community is focused especially on navigation attributes like length of navigable roads, the number of turn restrictions, signposts, and many more, in total 22 of such metrics are available. You can check it out at https://metrics.improveosm.org

About the data

Metrics are computed weekly and should be available on the portal at the end of each week. Metrics are generated for the whole world using as input the planet pbf downloaded from the official mirrors made available by the OSM community

Metrics are available starting with 8^th February 2016. In the top left corner, you can choose to see them by week, by month, or by quarter. We also have a nice feature for all OSM enthusiasts! For each metric in the left menu, you have a small info button where you see exactly what the metric means: complete description and the rules we applied when computing them, which tags were used if we counted ways, nodes or relations, etc.

How do we do it?

The platform was built using Apache Spark. Using big data technologies enabled us to have metrics for the whole world: on countries, states, counties, and a few metropolitan areas (metros are available only in North America for now). In order to use Apache Spark, we had to convert pbf to parquet first, so we achieved this using a packetizer that is open source and can be found here. After we have the parquets, using Spark’s DataFrame API we managed to have these metrics available in just a couple of hours.

We have also made the latest parquet files available for general use here.

If you have any suggestions or feedback, please do not hesitate to contact us. You can find details in the About section.

Happy mapping!

Working with ImproveOSM Data Dumps

Our ImproveOSM pipeline produces a pretty impressive number of suggested roads missing from OSM, missing oneway tags, and missing turn restrictions, based on analysis of billions of GPS data points. We make the results available as frequent data dumps in CSV format. In this post, I want to look at a way to integrate this data into your OSM mapping workflow.

If you just want to see ImproveOSM data in JOSM wherever you are currently mapping, you can just use the ImproveOSM JOSM plugin. For advanced users who want more flexibility, or who want to use this data in different ways, this post offers some guidance.

The data dumps are available from here. For this example, I will work with the most recent Direction of Flow data file. This highlights ways with potential missing oneway tag. After downloading and unzipping it, you will have a CSV file of about 16.5 megabytes that looks like this:

wayId;fromNodeId;toNodeId;percentage;status;roadType;theGeom;numberOfTrips
148617028;1867720648;89191396;99.5378927911275;SOLVED;THROUGHWAY;LINESTRING(2.217821 48.922613,2.217719 48.922618,2.217408 48.922633);1082
33555379;322840377;322840383;98.6301369863014;INVALID;LOCAL_ROAD;LINESTRING(4.999815 47.34294,4.999957 47.343062,4.999965 47.34315);146
17271190;178942503;2341050872;100;OPEN;LOCAL_ROAD;LINESTRING(11.070503 50.139245,11.070525 50.139213,11.070616 50.139099,11.070693 50.139032);74
.....

Since the theGeom field is in WKT, you can import it as a layer in QGIS pretty easily. Let’s fire up QGIS (I use 2.18) and add a Delimited Text layer.

In the dialog, select the downloaded CSV file as the file source. Set the delimiter to semicolon. QGIS detected for me that the geometry was in the theGeom field, and of type WKT, but you can set that manually if needed:

Upon clicking OK, QGIS wants us to define which CRS the coordinates are defined in. Select WGS84.

Now, we have a layer of line geometries that correspond to OSM ways that may be missing a oneway tag.

To make the file more manageable, let’s limit our selection to one country. I get country boundaries from Natural Earth (a fantastic resource!). After adding the country borders to QGIS, I can perform a spatial query. Before you do this, select the country you are interested in. I pick Mexico as an example.

Bring up the Spatial Query window. If you don’t see this menu item, you will need to enable the Spatial Query plugin.

Select the ImproveOSM layer as the source, and the Natural Earth layer as the query layer. Make sure to check the ‘1 Selected geometries’ checkbox, so we limit our query to Mexico.

The matching features will now be selected in the ImproveOSM layer. Make sure that layer is selected in the Layers Panel before you select Layer -> Save As.. from the QGIS menu. In this dialog, choose GeoJSON as the output type. Select a destination filename. Make sure that the CRS is set to WGS84. Make sure the ‘Save only selected features’ is checked, and Save.

Now you have a GeoJSON file with all OSM way geometries that may need a oneway tag. You can load this file into JOSM, using its GeoJSON plugin. To organize your work going through these, I would recommend using the Todo plugin and add the GeoJSON features to the todo list.

Detecting Traffic Signs in OpenStreetCam

OpenStreetCam’s mission is to help you improve OSM with street-view imagery. Photos taken with regular smartphones seem to be good enough for capturing map features like traffic signs, lanes or crosswalks. However, browsing the 120 million+ photos in OSC to find relevant things to map will take a while. The human factor is fundamental to OSM’s culture and we don’t see that changing, but we want to make editing street related attributes more efficient with automation.

We’re happy to announce a beta release of the traffic signs recognition on OpenStreetCam photos, made possible with machine learning. We processed a few million photos and detected around 500.000 traffic signs so far, currently available for tracks in several areas in United States and Canada. We’re working on extending the training sets and optimize the processing so that the area’s soon expanded.

What’s new from a user perspective: the track page on openstreetcam.org will now show detected traffic signs when available:

There’s a preview list of all detections in the track, detection overlays on photos and, of course, filters. Filters might now get a rep as something really exciting, but we’re excited about one of ours: the OSM status. Here’s why: after detecting a sign we compare it to the corresponding OSM feature and check if they’re consistent. Based on that, filtering is available.

For a practical example, let’s take speed limits: Instead of manually cross checking every detection with the maxspeed tag in OSM, one can only review detections where presumably maxspeed is not set or the value’s different in OSM. Just tick the Need review in OSM box.

Here are a few more examples of trips that have already been processed with our sign detections.

What’s next?

We’re busy working on a few things:

Scale the training sets and pipeline to extend the supported areas.
Traffic signs integration in the JOSM plugin.
Tagging new traffic signs support in the webpage.

If you like what we do and want to help:

First and foremost, you can use detections to improve OSM. If you’re seeing detections on tracks check them out, see what needs reviewing in OSM and edit. You can open iD or JOSM to photo’s location straight from the webpage.
Help us improve the traffic signs recognition. There’s a chance you will find some bad detections. You can review them and flag whether they’re good or bad, see the two buttons above the photo. We’re adding those reviews to training sets to improve recognitions, so please play nice.
Help us add these detections to the iD editor as well.

Tip: you can navigate between detections with Ctrl/Cmd + right/left arrows and confirm/invalidate with Ctrl/Cmd + up/down arrows. Goes pretty fast.

Talk to us, tell us what you like and what you don’t at openstreetcam@telenav.com or on Github.

Cygnus – conflation at your fingertips!

This is a follow-up blog post after the State Of the Map US 2017 conference held in Denver.

The process of conflation in GIS is defined as the act of merging two data layers to create one layer containing the features and attributes of both original layers.

Cygnus is a tool that compares external data with OSM, giving you a result file in JOSM XML format with all the changes. The comparison is made in a non-destructive way, so no OSM ways are ever deleted or degraded.

Workflow

NOTE – The license compatibility between the local data file and OSM has to be taken into account before adding anything in OSM. Also, please follow the OSM import procedures if you are planning to add external data to OSM.

First of all, you need to have a shapefile with local data in WGS84 spatial reference. This shapefile has to be filtered in different ways, depending on the tags you want to compare. For example, if you want to compare oneways, make sure to have a flow-direction/oneway/etc. attribute in the shapefile.

Translation

The first thing that has to be taken care of is to assure a proper attribute translation. I created a simple example for this exercise. I don’t want to get neck-deep in too many technical details so the main focus remains the process as a whole. I kept the attribute information for this example straightforward:

In order to create an OSM file from this data, I wrote a simple translation file that will be used together with ogr2osm.

Next, run the below command to obtain the OSM file.

python ogr2osm.py simple_streets.shp -t simple_translation.py -o simple_output.osm

Finally, I converted the OSM file to PBF using osmosis, because Cygnus requires a PBF file as input.

Cygnus goes to work!

Now that you have gone through the pre-processing of the local data file, we can offer it to Cygnus for processing. Note that your upload needs to be small-ish – the spatial extent needs to be smaller than 50×50 km and the file needs to be 20MB or smaller in size.

The interface of the Cygnus service is very simple – there are just two pages:

the home page where you add new jobs
the job queue page where you can see your progress and download the result

If your input file was uploaded successfully, Cygnus will go to work. Your job will be added to the back of the queue. When it’s your turn, Cygnus will read your PBF file, and download the OSM data for the same extent, using Overpass API. It will then compare your upload with the existing OSM data and produce the output file that you can download from the job queue.

NOTE – Everyone’s jobs are listed here, so be careful not to touch other users’ stuff.

Process the output in JOSM

Once Cygnus gives us the output, we can open it in JOSM and inspect it. This is by far the most important, and time consuming, step. Even though Cygnus does a best effort to connect ways where needed, it acts conservatively so it will not snap ways together that do not belong together.

Here are a few ways that got properly connected to the existing highway=secondary:

But there are situations where the distance was too far so Cygnus did not snap:

In this case, you need to manually connect the ways if that is appropriate.

When you are finally satisfied with your manually post-processed conflation result, you can go ahead and merge it with the OSM data and upload it!

Fire up the editors: ImproveOSM updated with many new things to fix in OSM

Our OSM team continually processes billions of anonymized GPS traces we receive through the Scout app and partners, in order to discover things potentially wrong or missing in OSM. We call this effort ImproveOSM, and it is a big part of Telenav’s overall mission to keep making OSM even better.

Missing Roads in Northern Brazil. The denser the GPS point cloud, the more trips and the more likely you are helping people get around more accurately!

Our most recent update to ImproveOSM was a particularly big one. In the last month, we added:

133 thousand missing roads tiles
- Another 75 thousand tiles that are likely parking areas or tracks
- Another 670 thousand (!) water tiles (see below)
300 thousand suspected turn restrictions with over 50% high confidence

Using ImproveOSM data

Perhaps you have not looked at ImproveOSM data before. It is available through the ImproveOSM website, which is based on the iD editor. The screenshots on this page are from that website. If you know how to edit with iD, you will find it easy to work with ImproveOSM data and use it to edit OSM. We wrote a post that goes into more detail a little while ago.

If you prefer JOSM, we have created an ImproveOSM JOSM plugin as well. it works similar to the website: you choose what ImproveOSM data you want to see (suspected missing roads, suspected wrong one-way roads, or suspected missing turn restrictions, or all of the above!), and the plugin will show you the ImproveOSM data as a separate layer. We also have a blog post about using the JOSM plugin.

Finally, a few interesting/funny examples of ImproveOSM data around the world.

ImproveOSM data points out that a new road alignment is now in use. Aerial imagery and OSM have not been updated yet. This is in northern Sweden.

Here, we stumble upon an under-mapped town north of Surat, India. Of course, there are un- and under-mapped areas everywhere in the world, but the ImproveOSM data shows that there are people driving around on these streets using a GPS-enabled app or vehicle — people who would benefit from better OSM data in their everyday lives. It is not hard to find places like this around the world.

Finally, an animation showing clusters of ‘water’ tiles. This is a side effect of the partner data we process. Since it’s anonymized there is no way to say anything about why these traces exist. Useful for OSM? Perhaps. Interesting? I think so!

Are you finding interesting, useful, funny, or wrong data in ImproveOSM? Let us know! Happy Mapping!

Mapping traffic signals and stop signs using MapRoulette

In our journey of improving OpenStreetMap, we are constantly searching for open-source data. This search is very important and is done before we start improving the map in a new area.

Currently, part of our team is focused on improving the Detroit area. So, before we started mapping we searched for useful geospatial data and we came across open data about traffic signals and stop signs for Wayne County, Detroit. The data can be found here and here.

We filtered out the traffic signals and stop signs that were already in OSM but there is still a significant amount of data that can be added in OSM. (912 – traffic signals and 8755 – stop signs). Due to this, we thought about creating a MapRoulette challenge.

About MapRoulette

MapRoulette is a micro-tasking tool used to fix bugs in OpenStreetMap and to improve it. A user can create tasks by uploading files that contain the location, ways, points with the error that has to be fixed, or files with features that are missing from the map and can be added by other users.

When creating a new task, the user gives specific instructions on what steps have to be followed to edit through this tool. Once a user has logged in, he can see on the map the created challenge and the pins which consist of tasks he can solve.

So, given the available data that we found, we created two challenges – one for traffic signals and the other for stop signs. Some general rules for mapping traffic signals and stop signs can be found on the OSM wiki – here and here.

Tags that we use for mapping

Stop signs – highway=stop
Traffic signals – highway=traffic_signal

Notes

If the traffic signal/stop sign is referring to all the highways entering the intersection, we add the traffic signal/stop sign at the intersection point.

If the traffic signal/stop sign is not referring to all the highways entering the intersection we add the traffic signal/stop sign before the intersection, where the sign/signal is positioned.
We need to add an additional tag if the road is bidirectional:
- for traffic signals, we use the traffic_signals:direction key with the forward or backward values to indicate the affected direction.

- for stop signs add direction=forward or direction=backward to indicate the affected direction.

The data has been published under Public Domain license.

Everyone who is keen on mapping is welcomed to help us.

Let’s improve OSM together!

More and Updated Data for ImproveOSM

ImproveOSM has been updated with many new roads. We processed recent GPS data from a number of data partners with some great results. A total of 30,000 new missing road tiles were added, over 17000 in Indonesia alone.

Aside from the missing roads, we added 67000 potential missing one-way roads that we detected with high confidence. Internal testing revealed only 6% false positives.

We are happy to continue providing OSM mappers with high-quality data about missing things in OSM based on billions of GPS traces. Because ImproveOSM is based on actual drives from people using navigation or mapping software in their vehicles, and we apply a pretty high threshold for the number of trips and quality of the GPS data, you can be pretty confident that every ImproveOSM feature will lead you to something you can add to OSM. Even if the aerial imagery is poor.

You should see the new data in your ImproveOSM plugin or on the ImproveOSM website very shortly. Happy mapping and let us know what you mapped using ImproveOSM!

Improve OSM adds missing roads in Guatemala

In a new data released today, we added about 500 tiles worth of missing roads in and around Guatemala!

Missing roads near Coatepeque, Guatemala in JOSM. Imagery from Bing.

We are excited to be adding more and more Missing Roads data to ImproveOSM using GPS data from our own users as well as from data partners like we did in Brazil and in this case.

You will notice that the tiles look a little different from the ones you are used to if you have used ImproveOSM before: they don’t show the individual points. This is because this particular data was processed a little differently. If you use JOSM, you will also see an update to the ImproveOSM plugin to accommodate this change.

While you are looking at the new Missing Roads, perhaps you will also notice some other recent improvements to the ImproveOSM website. We re-ran all tiles based on new map data from mid-April, and we improved our turn restriction detection so we won’t show a missing turn restriction when OSM already has an ‘only straight on’ restriction.

Happy Mapping!