Summer Dispatch From The Telenav Map Team

It has been an exciting summer! Besides our regular work, there was the annual State of the Map conference that we were all really looking forward to. We launched a new ImproveOSM web site. OpenStreetCam dash-cams are distributed to OSM US members. And more. Read all about it in our Summer Dispatch below!

State of the Map

Quite a few of us got to go to State of the Map in Milan, Italy! Our team hosted four presentations at the conference, and we are really happy with the interest and feedback we received. We made a lot of new map friends as well!

All SOTM presentations were recorded and posted on YouTube, so if you missed any of us, you can watch the presentations at your leisure:

Alina and Bogdan presenting our Machine Learning stack at SOTM 2018

We also had a booth at the conference where we talked about ImproveOSM and OpenStreetCam, and where 6 lucky winners received a Waylens OpenStreetCam dashboard camera!

Excited crowd right before one of the Waylens cameras is being given away!

Mapping

We continue to map in Canada, the United States, and Mexico. As always you can track our work on GitHub. We have been focusing a lot on adding missing road names for the larger metropolitan areas in the US. Our typical workflow is to identify local government road centerline data sources, verify the license, process them with Cygnus to find changed / new names, and manually add the names if we can verify them.

Local road centerline data the team identified in Colorado

We are excited that the US community is looking to build an overview of available road centerline databases from (local) governments. We hope the ones we identified can help bootstrap this initiative.

We also published some MapRoulette challenges around this topic. 

ImproveOSM

Right on time for State of the Map, we launched a complete redesign of improveosm.org, our portal for everything Telenav❤️OSM. The new site gives you quick access to our OSM initiatives, data and tools. Check it out!We also released more than 20 thousand new missing roads locations. These are added to the existing database of currently more than 2.4 million missing road locations. An easy way to start editing based on these locations is to download the ImproveOSM plugin for JOSM.

Locations of the new Missing Roads locations

OpenStreetCam

The steady growth of OpenStreetCam continues. Almost 4.5 million kilometers of trips are in the OSC database. This amounts to about 165 million images!

We started a collaboration with OpenStreetMap US to run a Camera Lending program. Through the program, OSM US members can apply to borrow a custom Waylens Horizon camera for up to three months. The camera captures high resolution images for OSC and uploads them automatically. Almost 20 mappers have a camera already, and they have driven about 30 thousand kilometers in the past couple of months!

The passenger’s seat of our Camera Man ToeBee, as he gets ready to dispatch a bunch of Waylens cameras

That’s a wrap for our summer dispatch folks! Thanks for reading and keep an eye on the blog for more from the Telenav Map Team. Be sure to follow us on Twitter as well @improveOSM and @openstreetcam. 👋🏼

 

Facebooktwittergoogle_plus

Map Metrics for OSM are now available

Telenav’s OSM team just released a portal where you can view different metrics on OSM.

Unlike other metrics views that are already available, this new tool for the OSM community is focused especially on navigation attributes like length of navigable roads, number of turn restrictions, signposts and many more, in total 22 of such metrics are available. You can check it out at https://metrics.improveosm.org

About the data

Metrics are computed weekly and should be available on the portal at the end of each week. Metrics are generated for the whole world using as input the planet pbf downloaded from the official mirrors made available by OSM community

Metrics are available starting with 8th February 2016. In the top left corner, you can choose to see them by week, by month or by quarter. We also have a nice feature for all OSM enthusiasts! For each metric in the left menu you have a small info button where you see exactly what the metric means: complete description and the rules we applied when computing them, which tags where used, if we counted ways, nodes or relations etc.

How do we do it?

The platform was built using Apache Spark. Using big data technologies enabled us to have metrics for the whole world: on countries, states, counties and a few metropolitan areas (metros are available only in North America for now). In order to use Apache Spark, we had to convert pbf to parquet first, so we achieved this using a parquetizer that is open source and can be found here.  After we have the parquets, using Spark’s DataFrame API we managed to have these metrics available in just a couple of hours.

We have also made the latest parquet files available for general use here.

If you have any suggestions or feedback, please do not hesitate to contact us. You can find details in the About section.

Happy mapping!

Facebooktwittergoogle_plus

Geohash JOSM plug-in

The Telenav OSM team has a new JOSM plugin for you: Geohash. This plug-in displays a layer on top of the JOSM map that contains the corresponding geohashes, up to zoom level 10. It also allows searching for a specific geohash and moves the map to the corresponding area.

Our team has been using this plugin internally and thought it may be useful for some of you as well.

This plug-in can be used by those who work in specific areas based on geohash units.

How to use the Geohash plug-in

The geohashes are automatically generated based on the user map view and zoom level. Increased zoom means increased depth level for geohashes.

To search for a geohash, use the Geohash plug-in dialog. Type or paste the geohash code in the text field and press the Search button. If the code is invalid, a message will be shown. Otherwise, the map view will be moved and zoomed over the selected geohash.

To clear generated geohashes, double click a geohash and all geohashes from his parent will be cleared. Also, right click on Geohash layer will show the option to ‘Clear geohashes’ that will leave only the depth one grid.

You can find the source code here: https://github.com/Telenav/geohash-plugin

We are looking forward to your feedback!

Facebooktwittergoogle_plus

Working with ImproveOSM Data Dumps

Our ImproveOSM pipeline produces a pretty impressive number of suggested roads missing from OSM, missing oneway tags, and missing turn restrictions, based on analysis of billions of GPS data points. We make the results available as frequent data dumps in CSV format. In this post, I want to look at a way to integrate this data into your OSM mapping workflow.

If you just want to see ImproveOSM data in JOSM wherever you are currently mapping, you can just use the ImproveOSM JOSM plugin. For advanced users who want more flexibility, or who want to use this data in different ways, this post offers some guidance.

The data dumps are available from here. For this example, I will work with the most recent Direction of Flow data file. This highlights ways with potential missing oneway tag. After downloading and unzipping it, you will have a CSV file of about 16.5 megabytes that looks like this:

wayId;fromNodeId;toNodeId;percentage;status;roadType;theGeom;numberOfTrips
148617028;1867720648;89191396;99.5378927911275;SOLVED;THROUGHWAY;LINESTRING(2.217821 48.922613,2.217719 48.922618,2.217408 48.922633);1082
33555379;322840377;322840383;98.6301369863014;INVALID;LOCAL_ROAD;LINESTRING(4.999815 47.34294,4.999957 47.343062,4.999965 47.34315);146
17271190;178942503;2341050872;100;OPEN;LOCAL_ROAD;LINESTRING(11.070503 50.139245,11.070525 50.139213,11.070616 50.139099,11.070693 50.139032);74
.....

Since the theGeom field is in WKT, you can import it as a layer in QGIS pretty easily. Let’s fire up QGIS (I use 2.18) and add a Delimited Text layer.

In the dialog, select the downloaded CSV file as the file source. Set the delimiter to semicolon. QGIS detected for me that the geometry was in the theGeom field, and of type WKT, but you can set that manually if needed:

Upon clicking OK, QGIS wants us to define which CRS the coordinates are defined in. Select WGS84.

Now, we have a layer of line geometries that correspond to OSM ways that may be missing a oneway tag.

To make the file more manageable, let’s limit our selection to one country. I get country boundaries from Natural Earth (a fantastic resource!). After adding the country borders to QGIS, I can perform a spatial query. Before you do this, select the country you are interested in. I pick Mexico as an example.

Bring up the Spatial Query window. If you don’t see this menu item, you will need to enable the Spatial Query plugin.

Select the ImproveOSM layer as the source, and the Natural Earth layer as the query layer. Make sure to check the ‘1 Selected geometries’ checkbox, so we limit our query to Mexico.

The matching features will now be selected in the ImproveOSM layer. Make sure that layer is selected in the Layers Panel before you select Layer -> Save As.. from the QGIS menu. In this dialog, choose GeoJSON as the output type. Select a destination filename. Make sure that the CRS is set to WGS84. Make sure the ‘Save only selected features’ is checked, and Save.

Now you have a GeoJSON file with all OSM way geometries that may need a oneway tag. You can load this file into JOSM, using its GeoJSON plugin. To organize your work going through these, I would recommend using the Todo plugin and add the GeoJSON features to the todo list.

Facebooktwittergoogle_plus

Fire up the editors: ImproveOSM updated with many new things to fix in OSM

Our OSM team continually processes billions of anonymized GPS traces we receive through the Scout app and partners, in order to discover things potentially wrong or missing in OSM. We call this effort ImproveOSM, and it  is a big part of Telenav’s overall mission to keep making OSM even better.

Missing Roads in Northern Brazil. The denser the GPS point cloud, the more trips and the more likely you are helping people get around more accurately!

Our most recent update to ImproveOSM was a particularly big one. In the last month, we added:

  • 133 thousand missing roads tiles
    • Another 75 thousand tiles that are likely parking areas or tracks
    • Another 670 thousand (!) water tiles (see below)
  • 300 thousand suspected turn restrictions with over 50% high confidence

Using ImproveOSM data

Perhaps you have not looked at ImproveOSM data before. It is available through the ImproveOSM web site, which is based on the iD editor. The screenshots on this page are from that web site. If you know how to edit with iD, you will find it easy to work with ImproveOSM data and use it to edit OSM. We wrote a post that goes into more detail a little while ago.

If you prefer JOSM, we have created an ImproveOSM JOSM plugin as well. it works similar to the web site: you choose what ImproveOSM data you want to see (suspected missing roads, suspected wrong one-way roads, or suspected missing turn restrictions, or all of the above!) and the plugin will show you the ImproveOSM data as a separate layer. We also have a blog post about using the JOSM plugin.

Finally, a few interesting / funny examples of ImproveOSM data around the world.

ImproveOSM data points out that a new road alignment is now in use. Aerial imagery and OSM have not been updated yet. This is in northern Sweden.

Here, we stumble upon an undermapped town north of Surat, India. Of course, there are un- and undermapped areas everywhere in the world, but the ImproveOSM data shows that there are people driving around on these streets using a GPS enabled app or vehicle — people who would benefit from better OSM data in their everyday lives. It is not hard to find places like this around the world.

Finally, an animation showing clusters of ‘water’ tiles. This is a side effect of the partner data we process. Since it’s anonymized there is no way to say anything about why these traces exist. Useful for OSM? Perhaps.. Interesting? I think so!

Are you finding interesting, useful, funny or wrong data in ImproveOSM? Let us know! Happy Mapping!

Facebooktwittergoogle_plus

More and Updated Data for ImproveOSM

ImproveOSM has been updated with many new roads. We processed recent  GPS data from a number of data partners with some great results. A total of 30,000 new missing road tiles were added, over 17000 in Indonesia alone.

Aside from the missing roads, we added 67000 potential missing one-way roads that we detected with high confidence. Internal testing revealed only 6% false positives.

We are happy to continue providing OSM mappers with high quality data about missing things in OSM based on billions of GPS traces. Because ImproveOSM is based on actual drives from people using navigation or mapping software in their vehicles, and we apply a pretty high threshold for number of trips and quality of the GPS data, you can be pretty confident that every ImproveOSM feature will lead you to something you can add to OSM. Even if the aerial imagery is poor.

You should see the new data in your ImproveOSM plugin or on the ImproveOSM web site very shortly. Happy mapping and let us know what you mapped using ImproveOSM!

Facebooktwittergoogle_plus

Collaboration brings nearly 1 million missing roads to ImproveOSM

If you go to ImproveOSM today, you will notice that it looks a lot different. No, we are not talking about the recent change to a completely iD-based editing environment, although that was pretty neat too J. We are talking about the massive increase in Missing Road tiles worldwide!

Missing roads everywhere!
Missing roads everywhere!

We added more than 800 thousand new road tiles to ImproveOSM all over the world. The anonymous GPS traces are sourced from INRIX, a company that provides traffic and connected car services. We are extremely excited to have such a huge boost to ImproveOSM and to OSM itself!

If you haven’t tried ImproveOSM recently, why not head over to improve-osm.org right now and explore the millions of missing roads, one-way streets and turn restrictions detected from big data analysis on anonymous GPS traces from drivers all over the world?

You can read more about the collaboration with INRIX in the joint press release.

Facebooktwittergoogle_plus

ImproveOSM now based on iD editor

With the help of ImproveOSM, Telenav’s project to analyze billions of GPS points to detect missing roads, one-ways, and turn restrictions, you have already looked at 60,000 missing road tiles, 15,000 one-way suggestions, and 2,000 turn restriction suggestions since the project launched in September 2015.

Today, the Telenav OSM team has released a completely new version of the ImproveOSM web site. ImproveOSM.org is now entirely based upon the OpenStreetMap iD editor. The new ImproveOSM combines the benefits of the familiar, user-friendly iD editing environment with the power of ImproveOSM detections.

The new ImproveOSM web site based on iD
The new ImproveOSM web site based on iD

The new ImproveOSM web site showing missing roads.

Since the new web site is based on iD, it should look very familiar and you should have little trouble getting started with it. The main difference you will see is that the ImproveOSM version of iD has a special panel, which shows ImproveOSM specific options, actions and information. If you have used ImproveOSM before, these will be familiar to you. You can mark items as solved or invalid and apply filters to determine which detections you see.

I do not want to go into too much detail in this post, but I do have a quick power tip: following up on many requests from you, you can now select multiple missing road tiles more easily by pressing shift and selecting one tile. This will automatically select all adjoining tiles within the current view.

Our goal is to integrate the ImproveOSM functionality into the main iD editor over time. To make that happen, your feedback is really important, so please do not hesitate to report bugs and ideas on the project GitHub page, where the source code will also become available soon.

We hope you enjoy the new ImproveOSM web site and look forward to your feedback! Happy mapping!

Facebooktwittergoogle_plus

Updates to ImproveOSM JOSM plugin for better usability

The team has been working on some nice updates to the ImproveOSM JOSM plugin. I have been taking the new version for a spin and wanted to report back.

In case you need a refresher: ImproveOSM is a suite of tools (currently a web site and a JOSM plugin) that takes the results of a massive data analysis comparing billions of GPS data points with existing OSM data and displays them in a way that makes it easy for any mapper to improve OSM with missing roads, turn restrictions, and one-way tags.

Missing Roads (red), One-ways (orange) and Turn Restrictions (blue) in the clustered view of the ImproveOSM JOSM plugin.
Missing Roads (red), One-ways (orange) and Turn Restrictions (blue) in the clustered view of the ImproveOSM JOSM plugin. This is the Dallas, Texas area. Imagery from Bing.

The improvements are fairly small but gave me a noticeably nicer workflow, so I thought it would be worth sharing.

The first improvement is that you can now right-click on any of the ImproveOSM layers in the layer panel to access the data filtering options for that layer.

Access the data filters using a right-click on the layer panel
Access the data filters using a right-click on the layer panel.

The data filters let you see part of the data for that layer based on various criteria, such as number of trips, confidence level, status and others. The criteria available vary by layer. Here is the filter window for Missing Roads, for example:

Filter window for missing roads

The filters themselves are not new, but you needed to go to the ImproveOSM panel to access them before. I think this is way quicker.

Another thing I really like is the improved visualization for the turn restrictions. The team made it much easier to see the from-via-to flow of the suggested restriction. The from-segment is now green and the to-segment is red. When selected, the info panel will also display more useful information than before:

The new visualization of the missing turn restriction. The 'from' segment is green, the 'to' segment is red.
The new visualization of the missing turn restriction. The ‘from’ segment is green, the ‘to’ segment is red.
The metadata we display for a turn restriction is now more relevant.
The metadata we display for a turn restriction is now more relevant.

The detailed info panel was improved for the other categories (missing roads and one-ways) as well.

Finally, when you are done mapping an ImproveOSM thing, you can now quickly mark the thing as invalid or solved, without having to enter a comment. We realized that this was not a very efficient workflow. You can still add a comment upon closing the issue, but now it’s easy to do it without, by right-clicking on the ‘solve’ or ‘invalidate’ buttons and selecting the appropriate action.

invalidate-context

These small but meaningful improvements made my work with ImproveOSM in JOSM much more efficient. We are always looking for more ways to make ImproveOSM better. If you have used ImproveOSM and you have a few minutes to spare, I would appreciate it if you filled out this survey. Thanks a lot!

Facebooktwittergoogle_plus

Improve OSM adds missing roads in Guatemala

In a new data release today, we added about 500 tiles worth of missing roads in and around Guatemala!

Missing roads near Coatepeque, Guatemala
Missing roads near Coatepeque, Guatemala in JOSM. Imagery from Bing.

We are excited to be adding more and more Missing Roads data to ImproveOSM using GPS data from our own users as well as from data partners, like we did in Brazil and in this case.

You will notice that the tiles look a little different from the ones you are used to if you have used ImproveOSM before: they don’t show the individual points. This is because this particular data was processed a little differently. If you use JOSM, you will also see an update to the ImproveOSM plugin to accommodate this change.

While you are looking at the new Missing Roads, perhaps you will also notice some other recent improvements to the ImproveOSM web site. We re-ran all tiles based on new map data from mid-April, and we improved our turn restriction detection so we won’t show a missing turn restriction when OSM already has a ‘only straight on’ restriction.

Happy Mapping!

Facebooktwittergoogle_plus