New zoom freezing feature for Geohash plugin

Built by Grab, the Geohash Java OpenStreetMap Editor (JOSM) plugin is widely used in map-making, but a common pain point is the inability to zoom in to a specific region without displaying new geohashes. Read to find out more about the issue and how the latest update addresses it.

Introduction

Geohash is an encoding system with a unique identifier for each region on the planet. Therefore, all geohash units can be associated with an individual set of digits and letters.

Geohash is a plugin built by Grab that is available in the Java OpenStreetMap Editor (JOSM) tool, which comes in handy for those who work on precise areas based on geohash units.

Background

Up until recently, users of the Geohash JOSM plugin were unable to stop the displaying of new geohashes with every zoom-in or zoom-out. This meant that every time they changed the zoom, new geohashes would be displayed, and this became bothersome for many users when it was unneeded. The previous behaviour of the plugin when zooming in and out is depicted in the short video below:

This led to the implementation of the zoom freeze feature, which helps users toggle between Enable zoom freeze and Disable zoom freeze, based on their needs.

Solution

As you can see in the image below, a new label was created with the purpose of freezing or unfreezing the display of new geohashes with each zoom change:

By default, this label says “Enable zoom freeze”, and when zoom freezing is enabled, the label changes to “Disable zoom freeze”.

In order to see how zoom freezing works, let’s consider the following example: a user wants to zoom inside the geohash with the code w886hu, without triggering the display of smaller geohashes inside of it. For this purpose, the user will enable the zoom freezing feature by clicking on the label, and then they will proceed with the zoom. The map will look like this:

It is apparent from the image that no new geohashes were created. Now, let’s say the user has finished what they wanted to do, and wants to go back to the “normal” geohash visualisation mode, which means disabling the zoom freeze option. After clicking on the label that now says ‘Disable zoom freeze’, new, smaller geohashes will be displayed, according to the current zoom level:

The functionality described above can also be illustrated in the following short video:

Another effect that enabling zoom freeze has is that it disables the ‘Display larger geohashes’ and ‘Display smaller geohashes’ options, since the geohashes are now fixed. The images below show how these options work before and after disabling zoom freeze:

To conclude, we believe that the release of this new feature will benefit users by making it more comfortable for them to zoom in and out of a map. By turning off the display of new geohashes when this is unwanted, map readability is improved, and this translates to a better user experience.

Impact/Limitations

In order to start using this new feature, users need to update the Geohash JOSM plugin.

What’s next?

Grab has come a long way in map-making, from using open source map-making software and developing its own suite of map-making tools to contributing to the open-source map community and building and launching GrabMaps. To find out more, read How KartaCam powers GrabMaps and KartaCam delivers comprehensive, cost-effective mapping data.

The year 2020 – packed with challenging events, strict circumstances, and with a tendency to redesign life as we know it. In spite of its unpredictable nature, 2020 has also marked a milestone for us, as it unveiled the long-overdue transition from OpenStreetCam to KartaView – an app that withstood a change in its aesthetic, but clearly not a change of heart.                                    

KartaView is still here for you, for the community – here to replenish the endless stream of OSM information, at a gradual but steady pace. 

How often did you find yourself editing and wished you had imagery to fall back on? Nothing beats precision mapping more than having a good quality recording from which you can extract valuable attributes. It’s even more fun when those images are recorded by you! 

If you’re eager to embark on this journey, you can grab it here for Android or iOS. Let’s do some exploring!

There are already more than 384 million images on KartaView, covering almost 8 million kilometers of road, so, obviously, getting started is easy enough that it can be done without much guidance. Throughout this blog post, we would like to sway you into making your first contribution to KartaView, whether it’s for personal or communal use. Our collection app still relies on its main functionalities, which you already know and love. 

Mounting – Invest in a good quality phone mount!

Carefully secure your phone by using an appropriate mount, to prevent imagery from being blurry or come across as shaky. Using a stable, well-positioned mount is essential for capturing top-notch street-level imagery. The phone should be as centered as possible so that attributes from both parts of the road can be captured during one ride. Use the camera preview in the app to check.

Record during good weather conditions!

Now that the phone is well mounted, let’s tap on that camera shaped icon before starting your engine. As soon as you begin, an assisted tilt will show up on the screen – it will help you straighten your phone’s position. You guessed it, it’s time to click on START! And STOP. As many times as your phone storage or microSD can take it.

Use the map to find new roads that are not yet collected. Remember: more new roads, more points!

Preview recordings

Time to Netflix and preview! Maybe there was some bad weather outside or some images got blurred. Once you finish recording, check the imagery and binge-watch your trip on fast-forward. 

Uploading

You can now either upload every recording or discard unwanted ones and upload the rest, it’s entirely up to you. Your upload speed may vary depending on your Internet connection.

Processing

Once your recordings reach our server, they will go through a complex procedure that involves snapping the photos to the roads, removing Personal Identifiable Information (faces and license plates), and preparing them to be published to kartaview.org. This is also when your points are calculated – the number you see during recording is mostly a ballpark figure that can give you a general idea of how many points you’ve earned. Keep in mind that your recordings might end up in a queue – don’t worry, they’ll be processed when they’re first in line.

Unmatched pictures are available as well, and they can be recognized as small dots on the map, while zooming in. 

Editing OSM

Having high-resolution and up to date KartaView imagery is a vital source in editing OpenStreetMap. It helps you edit new features (e.g. lanes, turn restrictions, road name) in the map or validate the existing ones, especially while doing armchair mapping. This is a discussion on which we’ll emphasise in upcoming posts, fully dedicated to editing. 

In drawing things to a close, we’d like you to know that KartaView isn’t just about street-level imagery – in fact, as this field is becoming more dynamic than ever, we too try to make the most out of this progression – whether it’s indoor mapping, disaster surveying or simply embedding the coordinates of your favorite pizza place, images are key. 

Walking, driving, hiking, biking are all KartaView friendly activities that could make OpenStreetMap a better and better map for the world!

How are you using KartaView?

GPS technology is growing fast along with the mapping industry. There are a lot of navigation apps on the market, trying to step forward using different technologies and map data. As you know, behind all its routing processes and algorithms, a good routing application must have a good quality map.

Our team, Telenav, is developing an Open-Street-Map (OSM) based app, called Scout. Why Open-Street-Map? In my opinion OSM is a good choice because it’s an open, fast growing and well built-up community map. With its great involvement and commitment, the community is constantly trying to keep all the map features from OSM up to date.

For the purpose mentioned above, Telenav decided to improve the quality of the OSM map features, used for navigation, in Phoenix, Arizona.

From September until now, the whole project workflow was realized in 3 main steps:

Step #1. Research and get in touch with local community (research on local open source data, keep in touch with local OSM users, official traffic signs and driving rules, research on HOV and toll roads, roads under construction, other specific road features of Arizona State).

Step #2. Build-up process:

1. Road geometry, Road Name, One Way, Gates and Other Geometric Feature (turning circles, turning loops, etc.)
2. Signpost
3. Turn Restrictions and Traffic Lights
4. Lane and Turn Lane Info, including HOV Lanes and Toll Roads
5. Speed Limit Editing

Step #3. Quality Assurance (QA for every feature edited by our team in OSM, some road name special issues, repairing broken relations, final edits and corrections based on Improve OSM plug-in, Osmose and Keep Right errors, tile-by-tile verification).

The first step was a very important one because the team had to get in touch with the local community and mapping rules. Also, we researched other open source and free of charge data beside the already well-known sources used by OSM users (TIGER Roads, Bing, Digital Globe, Open-Street-Cam, Mapillary). The under-construction roads found in this step were permanently monitored, to be edited later.

The second step was the most time consuming and it included the editing and reviewing of all the most important navigable OSM features, starting from motorway, trunk, primary, secondary, tertiary, residential and service ways. The whole editing process lasted for about 3 moths and the workflow was mainly based on a tile-by-tile method. We also edited based on way category or route reference. During this period, we helped the community to increase the OSM quality as you can see below:

The third step was the last but not the least important. Because we wanted to have good quality features in OSM, we had to make a closure check on our edits, already made in the build-up process and where it was necessary, to fill the gaps. So, we used different QA procedures, queries, other plug-ins, error identifiers and even tile by tile verification.

For the queries based QA, we ran some predefined scripts based on OSM datasets, using pgAdmin and PostgreSQL. The queries were aimed mainly at finding:

• Wrong one-way direction
• Wrong number of lanes
• Untagged roads
• Long relation members
• Turn Restrictions with unusual number of members
• Ramp has name or reference number
• Similar names and destinations
• Detect possible roundabouts, etc.

The main JOSM plug-in used to complete the missing roads was Telenav ImproveOSM (also available on https://improveosm.org). The plug-in focuses especially on missing roads, one ways and turn restrictions.

Also, a good source of errors to be checked can be found on these error identifier sites: http://osmose.openstreetmap.fr/en/map/ and https://keepright.at/.

Doing the QA, we had an important overview of all our edits, especially of our wrong ones. In this step, we resolved also a name tag issue we’d came across during step #2. Step #3 represents confidence of a good quality editing process for the Telenav team.

When we needed a double check of our work, the community was a good help, giving us important feedback. For the next months, we’ll surely keep in touch continuously with the local OSM users, looking forward for their feedback to keep up a good maintenance work. The top editing users were the most helpful users, giving us precious and pertinent feedbacks.

In the GIF below you can see an evolution of Telenav mapping team edits during the last moths, in Phoenix, Arizona. Darker colors indicate high density and brighter colors low density.

Looking from a map analyst’s point of view, turn restrictions are some of the most important features a map can have. Turn restrictions influence a lot the way a route is made. If they are wrongly edited, they can cause bad routing, having big consequences on travel time, travel directions, maneuvers, and so on. That’s why we decided to talk a little bit more about this case: editing turn restrictions.

We worked on this issue for 3 months, starting from November 2017. We succeeded to review a large number of intersections between motorway, trunk, primary, secondary, tertiary, residential, and service ways from the Phoenix area.

During our project we accomplished:

• to add new turn restrictions
• to correct some of the damaged ones
• to remove some of the wrong ones.

The main sources of adding or editing turn restrictions were open-source:

• Satellite imagery: Digital Globe, Bing, Esri World Imagery
• Street-level imagery: Open-Street-Cam (OSC), Mapillary.

The software used in editing OSM was JOSM, an extensible editor for Open-Street-Map for Java 8. Also, for better visualization, we used different Map Paint Styles (MapCSS) and thematic Layers (Open-Street-Cam, Mapillary, Mapillary object layer).

With a self-developed procedure, we identified all that could be a turn restriction sign in all the OSC track photos, overlapping our area. In the first part of our project, we looked over 1723 turn restriction signs identified in Open-Street-Cam, spatially distributed as in the map below. Before adding any turn restriction, every detection or sign found in Open-Street-Cam was first validated based on the open sources first mentioned.

In the second part, we managed to identify turn restriction signs also based on Mapillary Imagery and Mapillary Object Layer.

All the reviewed turn restriction signs summed in the end 3289, from which 350 turned to be missing from Open-Street-Map. During this task, we also reviewed 11932 and added 376 new traffic lights.

The final step for our task was to do some QA testing. Firstly, we used a query-based QA to verify the quality of all turn restrictions from the OSM Phoenix area. For this, we ran some predefined scripts based on OSM datasets, using pgAdmin.

The queries aimed mainly:

• Long relation members
• Turn Restrictions with an unusual number of members
• “Odd” tagging in existing turn restrictions
• Conditional turn restrictions with an old tagging scheme

Example of a query used to identify unusual turn restrictions from OSM:

We wanted to make sure we’re covered all turn restrictions. So, we used the Telenav ImproveOSM plug-in in JOSM. The plug-in highlights the possible turn restrictions locations based on road geometries and other probe data.

This kind of errors identified by us during the query-based QA can also be found on http://osmose.openstreetmap.fr/en/map/ and https://keepright.at/ sites. These two sites contain datasets with all kinds of errors from OSM. The datasets were downloaded, extracted using some Python scripts, clipped after our bounding box, and filtered using some SQL queries in pgAdmin. The results were exported and compared with the initial errors that resulted in the first QA step and where it was necessary, the turn restrictions were modified or corrected.

The heat map below presents all the turn restrictions edited by the Telenav team, from November until now, in Phoenix. Darker colors indicate high density and brighter colors low density.

It has been only two years since the Latin American OSM community decided it was necessary to have a regional event. The first event took place in Santiago de Chile and the second one in Sao Paulo, Brazil. At the end of the second edition, there were a few places considering organizing the third edition of the event. When I found out Lima was chosen I was very happy. Lima is one of the cities you never get tired to visit and you always discover new things, especially new amazing food 😉

This year Telenav had the chance to be a sponsor. Every State of the Map is a gathering of different efforts that combining together keeps improving the data of the largest geospatial database in the world. This sponsor supported the attendance of Juliana Hernández (Peasent mappers- Tools for learning, teach and strengthen cartography) and Daniel Quisbert (Installing and configuring Offline OSM in Linux).  Both conferences were highly appreciated by the attendees.  Also, Mapbox and Telenav co-hosted the first anniversary of Geochicas, it was a lovely evening in a historic bar from Lima in the one new geochicas from Perú and Colombia joined the group. We expect more companies to collaborate together to reduce the gender gap in OSM. There are many ideas for what Geochicas will be doing in 2018, please check the full information here.

The keynote was given by Philipp Kandal (A journey with OSM from the past into the Future) In the one Philipp shared his experiences over the years with OSM data and how probe data and using machine learning tools the map its being improved in a tremendous way.

From my side, I had the chance to show the mapping efforts in Canada and the preliminary mapping results from Ecuador, a pilot project we are currently developing to improve road data in one LATAM country. By using Satellite imagery and OpenStreetCam data we started improving the OSM data inroads in September this year. Find the slides here. So far the results we have are the following:

Stay tuned for the updated results in the following months!

This is just the beginning of future collaborations Telenav would like to keep doing with Latam communities by providing tools and guidance in previous projects we have done.

Thanks to the organization team and attendees of SOTM LATAM 2017, see you in other places in 2018!

This article was written by Adrian Bona as a draft for a talk at State of the Map US in Boulder, Colorado this past month. The talk did not make it into the program, but the technology lives on as a central part of our OpenStreetMap technology stack here at Telenav. We will continue to deliver weekly Parquet files of OSM data. Adrian has recently moved on from Telenav, but our OSM team is looking forward to hearing from you about this topic! — Martijn

Getting started with OpenStreetMap at a large scale (the entire planet) can be painful. A few years ago we were a bit intrigued to see people waiting hours or even days to get a piece of OSM imported in PostgreSQL on huge machines. But we said OK … this is not Big Data. Meanwhile, we started to work on various geo-spatial analyses involving technologies from a Big Data stack, where OSM was used and we were again intrigued as the regular way to handle the OSM data was to run osmosis over the huge PBF planet file and dump some CSV files for various scenarios. Even if this works, it’s sub-optimal, and so we wrote an OSM converter to a big data-friendly columnar format called Parquet. The converter is available at github.com/adrianulbona/osm-parquetizer.Hopefully, this will make the valuable work of so many OSM contributors easily available for the Big Data world.

How fast?

Less than a minute for Romania-latest.osm.pbf and ~3 hours (on a decent laptop with SSD) for the planet-latest.osm.pbf.

Getting started with Apache Spark and OpenStreetMap

The converter mentioned above takes one file and not only converts the data but also splits it in three files, one for each OSM entity type – each file basically represents a collection of structured data (a table). The schemas of the tables are the following:

node
 |-- id: long
 |-- version: integer
 |-- timestamp: long
 |-- changeset: long
 |-- uid: integer
 |-- user_sid: string
 |-- tags: array
 |    |-- element: struct
 |    |    |-- key: string
 |    |    |-- value: string
 |-- latitude: double
 |-- longitude: double

way
 |-- id: long
 |-- version: integer
 |-- timestamp: long
 |-- changeset: long
 |-- uid: integer
 |-- user_sid: string
 |-- tags: array
 |    |-- element: struct
 |    |    |-- key: string
 |    |    |-- value: string
 |-- nodes: array
 |    |-- element: struct
 |    |    |-- index: integer
 |    |    |-- nodeId: long

relation
 |-- id: long
 |-- version: integer
 |-- timestamp: long
 |-- changeset: long
 |-- uid: integer
 |-- user_sid: string
 |-- tags: array
 |    |-- element: struct
 |    |    |-- key: string
 |    |    |-- value: string
 |-- members: array
 |    |-- element: struct
 |    |    |-- id: long
 |    |    |-- role: string
 |    |    |-- type: string

Now, loading the data in Apache Spark becomes extremely convenient:

val nodeDF = sqlContext.read.parquet("romania-latest.osm.pbf.node.parquet")
nodeDF.createOrReplaceTempView("nodes")

val wayDF = sqlContext.read.parquet("romania-latest.osm.pbf.way.parquet")
wayDF.createOrReplaceTempView("ways")

val relationDF = sqlContext.read.parquet("romania-latest.osm.pbf.relation.parquet")
relationDF.createOrReplaceTempView("relations")

From this point on, the Spark world opens and we could either play around with DataFrames or use the beloved SQL that we all know. Let’s consider the following task:

For the most active OSM contributors, highlight the distribution of their work overtime.

The DataFrames API solution looks like this:

val nodeDF = nodeDF
    .withColumn("created_at", ($"timestamp" / 1000).cast(TimestampType))
    .createOrReplaceTempView("nodes")

val top10Users = nodeDF.groupBy("user_sid")
    .agg(count($"id").as("node_count"))
    .orderBy($"node_count".desc)
    .limit(10)
    .collect
    .map({ case Row(user_sid: String, _) => user_sid })
    
nodeDF.filter($"user_sid".in(top10Users: _*))
    .groupBy($"user_sid", year($"created_at").as("year"))
    .agg(count("id").as("node_count"))
    .orderBy($"year")
    .registerTempTable("top10UsersOverTime")

The Spark SQL solution looks like this:

select 
    user_sid, 
    year(created_at)) as year,
    count(*) as node_count
from 
    nodes
where 
    user_sid in (
        select user_sid from (
            select 
                user_sid, 
                count(*) as c 
            from 
                nodes 
            group by 
                user_sid 
            order by 
                c desc 
            limit 10
        )
    )
group by 
    user_sid, 
    year(created_at)
order by 
    year

Both solutions are equivalent, and give the following results:

Even if we touched only a tiny piece of OSM, there is nothing to stop us from analyzing and getting valuable insights from it, in a scalable way.

If you are curious about more advanced interaction between OpenStreetMap and Apache Spark, take a look at this data bricks notebook.

OpenStreetMap Parquet files for the entire planet?

Telenav is happy to announce weekly releases of OpenStreetMap Parquet files for the entire planet at osm-data.skobbler.net.

New ImproveOSM missing road tiles are available! The new data is very helpful as they can help you to target the missing roads, add them to OSM, and thus greatly improving the map.

Worldwide, there are 113048 new road tiles.  The countries with the highest number of tiles are Russia – 38669 tiles, United Kingdom – 8890 tiles, Kazakhstan – 10993 tiles, India –  9418 tiles, and the United States- 7560 tiles (see graph below). There are few new tiles in Detroit too so that you are welcome to give us a hand with them! You can find more information about our work in Detroit on our blog (http://blog.improveosm.org/en/2017/08/lane-number-and-turn-lane-editing-in-detroit/).

Since we started editing in Detroit, we focused on making OSM navigation ready. We started with the basics: road geometry, road name, turn restrictions, and then we were able to further build on this foundation by adding details like lanes and turn lanes. In the last four months, we focused on adding and updating the lane info (lane number and turn lane) on motorway, motorway_link, trunk, trunk_link, primary, primary_link, secondary, secondary_link roads in Detroit, Michigan.

For editing lanes and turn lanes we used JOSM, the TurnLanes-tagging Editor plugin and the Lane and road attributes map paint style.

We had two kinds of lane editing: unidirectional road editing, bidirectional road editing. The only difference between those two is the direction tag used in the second case, as you can see in the below table:

For every edited case, we used a simple workflow:

• we split the way where the number of lanes changes
• we checked and double-checked the aerial imagery to make sure we enter the correct number of lanes and add the appropriate lanes tag
• we opened the turn lanes-tagging plugin and activated the Lane and road attributes map style
• using the plugin, we selected the type of the road: Unidirectional road or Bidirectional road
• we marked the number of lanes for each way needed
• we marked  the direction on each lane
• before uploading the data, we checked again that the turn lanes that we had added were similar to the markings on the road!
  

The approach of the main cases we’ve met during our edits is exemplified in the next GIFs.

Editing the number of lanes

Adding both ways lane

In some particular cases, when there were doubts, we consulted the OSM community on Github and Talk-US.

While editing, we paid special attention to other already existing features (like route relations, turn restrictions, speed limits, etc). Because all Telenav Mapping team was involved in this project, we established from the beginning some rules, in order to have consistency in our edits:

• Add a new lane only when you have a line marked on the road (use the satellite imagery, OSC photos to validate the marks).
• Links without any marks on-road or without one-way tag should be edited as a bidirectional road, adding one lane on both driving directions.
• Never add the turn lane before or after the continuous line mark on the road. The turn lane will be added starting from the beginning of the continuous line mark on the road.
• We split and edit lane numbers even when we have small segments of ways.
• The location of the junction nodes should be at the beginning of the continuous line marks.
• We always add the yellow both-way lane.
• We DO NOT add the yellow striped lanes and double marked line lanes.

The main sources used during the project were aerial imagery (Bing, Mapbox, NAIP, Digital Globe) and street-level imagery: OSC, Mapillary.

We worked on this issue for 2 months and succeeded to review a large part of the motorway, trunk, primary and secondary roads from the Detroit area, in order to add or update lane info. During this project, we managed to review 3100 miles and edit 1730 miles of roads.

Here’s how the number of miles of roads with lane information has increased during the project:

The edits we made cover a large area of the Wayne, Macomb and Oakland counties. In the GIF below you can see an evolution (difference between March and July) of our lane info edits in OpenStreetMap.

Heatmaps with our edits during the last four months:

When we finished editing lanes and turn lanes in Detroit, we started assessing the general quality of the lane info by using different approaches. Internally, we call this process quality assurance and we think it is vital to do it after the end of each project.

During the QA process, we edited lane info on about 400 miles of roads, and the main issues that we corrected were:

• incorrect number of lanes and turn lanes
• duplicated/overlapping ways
• missing both way lane
• oneways with lanes:forward/lanes:backward info
• check roundabouts to have the proper number of lanes

Below you can see some examples of our improvements:

Ever since we started our mapping project in Canada, nearly 8 months ago, we’ve been continuously working on bringing the OSM data to the level where all elements needed for routing get as detailed as possible.

Whether we are talking about the basics of road networks such as geometry, naming, or traffic flow direction, to in-depth details like the number of lanes, turn lanes, turn restrictions, signposts, and even complex relations referring to highways, we edit everything.

Our main focus is oriented towards the Top 5 metro areas: Toronto, Montreal, Ottawa, Vancouver, Calgary. These are the places where we spent most of our time researching for open data, adding new features, editing existing ones. In order to make sure that the overall state of OSM throughout the entire region of Canada is in a navigable ready state, we’ve also included the first 50 cities based on population.

So, let’s see some numbers and graphs because everybody likes those. If we start looking at the numbers for the entire region we can see a significant rise in road geometry that was added, around 3% (25,330 miles) out of the total numbers of miles. The same goes for roads that previously did not have name tags with a rise of a little over 3.5% (16,799 miles).

A more significant change can be noticed for features that weren’t extensively mapped before in the area, such as turn restrictions rising from 5254 to 54891, or signposts that hadn’t been mapped under the same standardized method. With the help of OpenStreetCam and Mapillary pictures, we’ve managed to add relevant signpost information increasing the number of nodes well over 68%.

If we break down the numbers for the Top 5 areas, the most noticeable changes can be observed for both Toronto and Montreal where one-way tags and signpost information have been improved.

One of our main goals is to focus not only on quantity but especially on quality. This is why we have multiple tools for integrity checking that are run periodically on the entire region of Canada. These tools cover a wide variety of cases that are being corrected weekly, such as road name flip-flops, unconnected ways, smoothness problems, misnamed roads, road names having their suffixes or prefixes abbreviated, and many more.

We make use of different QA tools (KeepRight/Osmose) to search and track issues in OSM that have either been added by mistake or have remained unedited after large imports. We’re also on the lookout to improve way accuracy and fix alignment issues.

Below you can see some examples of our improvements.