A fresh Cygnus+ version

Cygnus is a tool that helps the community improve the OSM map by comparing it with an external (local) map, and detecting differences between the two maps. The differences detected include road geometry and certain tags of interest (e.g.: ‘name’, ‘oneway’, ‘ref’, ‘maxspeed’) that are missing or have different value from OSM.

In this latest version of Cygnus, we have introduced new features and made a few overall improvements.

New Features

Minimum Way Length Distance

A new setting was added in this latest version: the minimum distance by which a way’s length in the OSM map may be extended using the external map.

Conflation setting which is defaulted to 15%. Maximum can be 50%

Extending a way’s length can be very useful where section of a road may be missing or a new road needs to be added. Existing ways will be extended when the local map has a significantly longer way (as defined by the maximum connect distance). If it is possible to connect the extended way to any new ways, this will be done automatically.

New Grouped Settings and Checkboxes

We have grouped settings for ease of use into conflation, way-related and tag-related groups.

New checkboxes were also created for adding/extending ways (to allow more refined usage), adding/changing tags and specifying a minimum lexical difference.

Conflation Results

The conflation zip file now contains a narration log detailing each change that is proposed by the conflation archive. The Cygnus+ output explains in the ‘telenav:action’ tag exactly what action was taken on each way (this same output goes into the narration log).

Improvements

The conflation algorithm was improved and also the readability of parameters on jobs in the queue.

We hope you benefit from those changes, and let us know if you have any questions or ideas for further improvements.

Facebooktwittergoogle_plus

OpenStreetCam – Confidence level visualization

The OpenStreetCam plug-in has once again brought a new feature to help its users. It now allows displaying and filtering aggregated detections based on their confidence level. The confidence level represents a measure of confidence of the validity of the detection or aggregated detection.

New aggregated detections display

Aggregated detections can now be displayed using a confidence level based color code. This is an optional behavior that can be turned on in the plug-in preference settings by selecting the following preference:

If selected, each aggregated detection will be displayed with a border of the corresponding color, according to the legend table displayed under the check-box.

Map display based on confidence level:

If the checkbox is not selected, all aggregated detections are displayed with a white border.

Confidence level filtering

With the introduction of the confidence level attribute, a new form of filtering was added. Filter data now allows setting a value for minimum and/or maximum confidence level. When set, only the aggregated detections with the appropriate confidence level will be displayed.

The confidence level minimum and maximum must be values between 0 and 1.
Facebooktwittergoogle_plus

ImproveOSM plugin – design updates

The latest version of the OpenStreetCam JOSM plugin has many design changes which make the usage more intuitive. The Telenav OSM team added new icons for the three main layers of the ImproveOSM plugin (Turn Restriction, Traffic Flow Direction, Missing Roads) and modified the cluster colors accordingly.

new icons in the layers’ panel
new icons in the menu

Also, some improvements were made regarding the visualization of turn restriction detections at a high zoom level. The color for the grouped detections was changed in order to match the aspect of the corresponding cluster and to facilitate spotting the detections from the map, the turn restriction icon was  changed too.

Facebooktwittergoogle_plus

OpenStreetCam JOSM plugin – new features

The Telenav OSM team just released a new version of the OpenStreetCam JOSM plugin. In the last couple of months we had improved our sign detections and improved the detection map view,  by displaying aggregated traffic sign detections instead of individual detections. Traffic signs are detected per OpenStreetCam photos, in dense areas the same traffic sign is detected several times. Checking several detected signs that represent the same traffic sign in real life is time-consuming and slows down the mapping process. In order to improve the mapping process, detections that represent the same traffic sign were aggregated into a single detection.

Map View Changes

Aggregated detections are displayed for high zoom levels starting with zoom level 16. Each aggregated detection is represented by a traffic sign cluster icon rotated based on the detection heading.

Aggregated detection data

An aggregated detection can be selected by a right click from the map and unselected by a double click.                                                                                        When an aggregated detection is selected the detection icon along with the belonging photo locations are highlighted on the map and its information is displayed in the right side OpenStreetCam Detection panel. The photo on which the detection has the best visibility is also loaded automatically.

Loading other photos that contains the detections can be done by either selecting a photo location from the map or by pressing the Next/Previous buttons from the OpenStreetCam Detection panel. For the associated shortcuts take a look at the OpenStreetCam shortcuts from JOSM Preferences.

By default, the plugin displays the photo locations of a selected cluster, but the plugin can be configured to display also the actual detections. This can be enabled from JOSM Preferences->OpenStreetCam plugin -> Aggregated detection settings.

If enabled individual detections are connected to the corresponding photo. This visualization is used mainly for debugging purposes.

Data filters

The OSC plugin data can be filtered based on various new filters. Some of these filters were already present and others were extended.

  • data to display – the type of data to be displayed for high zoom levels; by default, we display OpenStreetCam photos and aggregated detections; if both detections and aggregated detections are selected then besides aggregated detections only the detections that do not belong to any aggregation are displayed
  • only mine – displays OpenStreetCam data contributed by the logged in user
  • not older than – filters out data based on a given timestamp
  • detection filters – are applied to detection/aggregated detection data
    • mode – is applicable only to individual detections; if set filters data based on detection mode; manual detections are detections that were manually marked on the OpenStreetCam photos, while automatic detections are detections recognized automatically by our platform
    • edit status  – is applicable only to individual detections; if set filters detections based on edit status (if the user had edited or not in OSM the detection)
    • OSM comparison – filters the data based on the status;  OSM comparison represents a status of a detection regarding its presence in OpenStreetMap; this filter is useful since the mapper can visualize only detections that need to added to the map
    • detection type – filter data based on detection type and subtype; this filter is useful if the mapper would like to focus on mapping only a certain type of signs

Default filter settings can be reset by pressing the “Reset” button.

If you are interested in the components used for aggregating the traffic sign detections check out the following presentation:  https://youtu.be/3I8lsIttqVA?t=290.

Upcoming features
The JOSM plugin is work on progress, we are working on improving the usability and plan to add new features from time to time. In the near feature, we plan to improve the detection mapping workflows.

We hope that you enjoy the new features! If you have ideas, suggestions or encounter any issue with the plugin during editing sessions please submit either to the GitHub issue page or to the Feedback forum .

Have fun improving the map by using OpenStreetCam images!


Facebooktwittergoogle_plus

ImproveOSM plugin – new features

The Telenav OSM team just released a new version of the ImproveOsm plugin.  We have added a location search box (helpful feature for reaching an area to be mapped) and a button for downloading previously selected ways.

Location search box

A new feature we added to the ImproveOSM plugin is the location search box.  It enables the user to visualize the desired geographical point at a higher zoom level after entering its coordinates in the text field.                     The input of the search box should contain the values for latitude and longitude separated by a comma. If the plugin doesn’t understand your input, an intuitive message is displayed in order to help you. This works similar to the JOSM Jump to position but the search box brings advantages such as the single field for data collecting and more validations for the latitude-longitude values.



Download way button

Another feature is the download way button for the Traffic Flow Direction layer. This enables the user to download the ways of the selected road segments in a new Data layer. This option is only available if the Traffic Flow Detection Layer is active. Furthermore, the button becomes enabled when one or more road segments are selected.

Facebooktwittergoogle_plus

Detecting Traffic Signs in OpenStreetCam

OpenStreetCam’s mission is to help you improve OSM with street-view imagery. Photos taken with regular smartphones seem to be good enough for capturing map features like traffic signs, lanes or crosswalks. However, browsing the 120 million+ photos in OSC to find relevant things to map will take a while. The human factor is fundamental to OSM’s culture and we don’t see that changing, but we want to make editing street related attributes more efficient with automation.

We’re happy to announce a beta release of the traffic signs recognition on OpenStreetCam photos, made possible with machine learning. We processed a few million photos and detected around 500.000 traffic signs so far, currently available for tracks in several areas in United States and Canada. We’re working on extending the training sets and optimize the processing so that the area’s soon expanded.

What’s new from a user perspective: the track page on openstreetcam.org will now show detected traffic signs when available:

There’s a preview list of all detections in the track, detection overlays on photos and, of course, filters. Filters might now get a rep as something really exciting, but we’re excited about one of ours: the OSM status. Here’s why: after detecting a sign we compare it to the corresponding OSM feature and check if they’re consistent. Based on that, filtering is available.

For a practical example, let’s take speed limits: Instead of manually cross checking every detection with the maxspeed tag in OSM, one can only review detections where presumably maxspeed is not set or the value’s different in OSM. Just tick the Need review in OSM box.

Here are a few more examples of trips that have already been processed with our sign detections.

What’s next?

We’re busy working on a few things:

  • Scale the training sets and pipeline to extend the supported areas.
  • Traffic signs integration in the JOSM plugin.
  • Tagging new traffic signs support in the webpage.

If you like what we do and want to help:

  • First and foremost, you can use detections to improve OSM. If you’re seeing detections on tracks check them out, see what needs reviewing in OSM and edit. You can open iD or JOSM to photo’s location straight from the webpage.
  • Help us improve the traffic signs recognition. There’s a chance you will find some bad detections. You can review them and flag whether they’re good or bad, see the two buttons above the photo. We’re adding those reviews to training sets to improve recognitions, so please play nice.
  • Help us add these detections to the iD editor as well.

Tip: you can navigate between detections with Ctrl/Cmd + right/left arrows and confirm/invalidate with Ctrl/Cmd + up/down arrows. Goes pretty fast.

Facebooktwittergoogle_plus

New Features and Enhancements in Cygnus+

Cygnus is the Telenav Mapping conflation tool. We use it a lot internally to compare approved external data sources with existing OSM data, but there is also a public version. We outlined how it works in an earlier blog post. In this post, I want to highlight some of the newer features in Cygnus. These new features are based on the feedback from our team of Map Analysts, who use the tool in their day to day work.

Discarding Very Short Segments

Cygnus outputs the differences in geometry between existing OSM data and the spatial data that we want to use to improve OSM. Sometimes, when the differences are very tiny, Cygnus used to export very short ways. These are not really meaningful enhancements, and clutter up the result data. Therefore, we implemented a length filter. Ways shorter than a defined length threshold will not be included in the output. Based on experience, we set the default to 5 meters. In the internal (command line) version our team uses, this can be tweaked using a parameter. In the public web version, this is not yet possible. We can consider adding it if there is sufficient demand.

An example of Cygnus in action. It finds an opportunity for improvement (possibly incorrect street name) as well as a false positive (degraded road geometry)

Road Names

When comparing road geometry, Cygnus not only compares geometry, but also road names. An annoying side effect we noticed is that road names are often not exactly the same in OSM as they are in the external data we compare with. This does not mean that the external data is necessarily better. For example, OSM could say that the name of a road is “River Road”, and the external data source could say it is “River Rd”. This is not a meaningful difference, and we would want to exclude those in most cases. So we added a string distance based  threshold in Cygnus to filter out similar strings. It is set to a sensible default which, again, can be tweaked in the command line version we use internally, but not yet in the web version.

Another Cygnus improvement related to road names is to ignore name differences on certain types of ways: roundabouts and service roads. Roundabout ways in OSM do not have names by convention, unless the roundabout itself has a name, so they should generally not be added. Service roads technically can have names in OSM, but it is not common. In external data, they do sometimes have names, but if they do, it usually does not make sense to add them to OSM. Based on our experience, they often have descriptive names like ‘driveway’ or ‘access road’ in the source data.

Using Cygnus

You can use Cygnus yourself by going to http://cygnus.improve-osm.org/ and uploading your source data file. You need to do a fair amount of work to prepare the source data: translating the source attributes into valid OSM tags, and converting to OSM PBF. And always remember to consider carefully what you do with the result. Cygnus is not designed to be an automated import tool. Every suggested change should be manually reviewed.

Let us know how you have used, or would like to use Cygnus!Facebooktwittergoogle_plus

Cygnus – conflation at your fingertips!

This is a follow-up blogpost after the State Of the Map US 2017 conference held in Denver.

The process of conflation in GIS is defined as the act of merging two data layers to create one layer containing the features and attributes of both original layers.

Cygnus is a tool that compares external data with OSM, giving you a result file in JOSM XML format with all the changes. The comparison is made in a non-destructive way, so no OSM ways are ever deleted or degraded.

Workflow

NOTE – The license compatibility between the local data file and OSM has to be taken into account before adding anything in OSM. Also, please follow the OSM import procedures if you are planning to add external data to OSM.

First of all you need to have a shapefile with local data in WGS84 spatial reference. This shapefile has to be filtered in different ways, depending on the tags you want to compare. For example, if you want to compare oneways, make sure to have a flow-direction/oneway/etc. attribute in the shapefile.

Translation

The first thing that has to be taken care of is to assure a proper attribute translation. I created a simple example for this exercise. I don’t want to get neck-deep in too many technical details so the main focus remains the process as a whole. I kept the attribute information for this example straightforward:

In order to create an OSM file from this data, I wrote a simple translation file that will be used together with ogr2osm.

Next, run the below command to obtain the OSM file.

python ogr2osm.py simple_streets.shp -t simple_translation.py -o simple_output.osm

Finally, I converted the OSM file to PBF using osmosis, because Cygnus requires a PBF file as input.

Cygnus goes to work!

Now that you have gone through the pre-processing of the local data file, we can offer it to Cygnus for processing. Note that your upload needs to be small-ish – the spatial extent needs to be smaller than 50×50 km and the file needs to be 20MB or smaller in size.

The interface of the Cygnus service is very simple – there are just two pages:

  • the home page where you add new jobs
  • the job queue page where you can see your progress and download the result

If your input file was uploaded successfully, Cygnus will go to work. Your job will be added to the back of the queue. When it’s your turn, Cygnus will read your PBF file, and download the OSM data for the same extent, using Overpass API. It will then compare your upload with the existing OSM data and produce the output file that you can download from the job queue.

NOTE – Everyone’s jobs are listed here, so be careful not to touch other users’ stuff.

Process the output in JOSM

Once Cygnus gives us the output, we can open it in JOSM and inspect it. This is by far the most important, and time consumig, step. Even though Cygnus does a best effort to connect ways where needed, it acts conservatively so it will not snap ways together that do not belong together.

Here are a few ways that got properly connected to the existing highway=secondary:

But there are situations where the distance was too far so Cygnus did not snap:

In this case, you need to manually connect the ways if that is appropriate.

When you are finally satisfied with your manually post-processed conflation result, you can go ahead and merge it with the OSM data and upload it!Facebooktwittergoogle_plus

Check out the new tutorial section

Recently we have launched a video tutorial section on improve.osm.org, to help our users get up to speed with our tools and let you know what are all the possibilities and all the versatility of our products for the community’

So far, we have 2 videos. One is focused on doing a gentle introduction to Improve OSM, some sort of a guided tour of the Improve OSM web site for new users. This is the perfect starting point if you’re new here.

The second one is a step-by-step guide to fixing your first one-way street in OpenStreetMap using the Improve OSM web site. You can see it here.

We’ll prepare more tutorial videos soon. Let us know what other aspects of our products you’d like to be featured in the future.Facebooktwittergoogle_plus

Improve OSM Tool Highlight: Cygnus

Improve OSM is all about making OpenStreetMap better with the help of billions of GPS points that users of Scout apps and others contribute automatically. This allows us to identify missing roads, one-way streets and turn restrictions that are not in OSM yet. However, our OSM engineers think about other ways to help make OSM better as well. Looking at what is going on in the OpenStreetMap community, I see a lot of groups and individuals struggle with the challenge of using external data in OpenStreetMap. Many governments now open up data that was closed before. With proper caution and preparation, we may be able to use this data in OSM.

The technical challenge here is to combine the new, external data with what is already present in OSM. In the GIS world this is called conflation. There are some tools out there to help with this task, but they are hard to use. So we started to think about this from a practical point of view and a specific use case: what if you have an external database of just roads, and you want to add only the new roads to OSM?

cygnus-osm-merge

We developed a tool that can do just that, and we called it Cygnus. I wrote about Cygnus on my OSM diary before, announcing it here and then writing about it being used by OSM user MikeN here. We added a link to the Cygnus web site to the ‘Other tools’ section of the ImproveOSM web site.

We made Cygnus as easy to use as possible, but it is still not straightforward. It should not be. Conflation is hard, and what you are doing is importing data into OSM, which should be done with the utmost caution! If you want to get started using Cygnus, please get in touch. We are happy to help!Facebooktwittergoogle_plus