Telenav presence in State of the Map Latam 2017

It has been only two years since the Latin American OSM community decided it was necessary to have a regional event. The first event took place in Santiago the Chile and the second one in Sao Paulo, Brazil. At the end of the second edition there were a few places considering to organize the third edition of the event. When I found out Lima was chosen I was very happy. Lima is one of the cities you never get tired to visit and you always discover new things, specially new amazing food 😉

This year Telenav had the chance to be sponsor. Every State of the Map is a gathering of different efforts that combining together keeps improving the data of the largest geospatial database in the world. This sponsor supported the attendance of Juliana Hernández (Peasent mappers- Tools for learn, teach and strengthen cartography) and Daniel Quisbert (Installing and configuring Offline OSM in Linux).  Both conferences where highly appreciated by the attendees.  Also Mapbox and Telenav co-hosted the first anniversary of Geochicas, it was a lovely evening in an historic bar from Lima in the one new geochicas from Perú and Colombia joined the group. We expect more companies collaborating together to reduce the gender gap in OSM. There are many ideas for what Geochicas will be doing in 2018, please check the full information here.

The keynote was given by Philipp Kandal (A journey with OSM from the past into the Future) In the one Philipp shared his experiences over the years with OSM data and how probe data and using machine learning tools the map its being improved in a tremendous way.

From my side I had  the chance to show the mapping efforts in Canada and the preliminary mapping results from Ecuador, a pilot project we are currently developing to improve road data in one LATAM country. By using Satellite imagery and OpenStreetCam data we started improving the OSM data in roads since September this year. Find the slides here. So far the results we have are the following:

Stay tuned for the updated results in the following months!

This is just the beginning of future collaborations Telenav would like to keep doing with Latam communities by providing tools and guidance in previous projects we have done.

Thanks to the organization team and attendees of SOTM LATAM 2017, see you in other place in Latam in 2018!

Facebooktwitter

Fire up the editors: ImproveOSM updated with many new things to fix in OSM

Our OSM team continually processes billions of anonymized GPS traces we receive through the Scout app and partners, in order to discover things potentially wrong or missing in OSM. We call this effort ImproveOSM, and it  is a big part of Telenav’s overall mission to keep making OSM even better.

Missing Roads in Northern Brazil. The denser the GPS point cloud, the more trips and the more likely you are helping people get around more accurately!

Our most recent update to ImproveOSM was a particularly big one. In the last month, we added:

  • 133 thousand missing roads tiles
    • Another 75 thousand tiles that are likely parking areas or tracks
    • Another 670 thousand (!) water tiles (see below)
  • 300 thousand suspected turn restrictions with over 50% high confidence

Using ImproveOSM data

Perhaps you have not looked at ImproveOSM data before. It is available through the ImproveOSM web site, which is based on the iD editor. The screenshots on this page are from that web site. If you know how to edit with iD, you will find it easy to work with ImproveOSM data and use it to edit OSM. We wrote a post that goes into more detail a little while ago.

If you prefer JOSM, we have created an ImproveOSM JOSM plugin as well. it works similar to the web site: you choose what ImproveOSM data you want to see (suspected missing roads, suspected wrong one-way roads, or suspected missing turn restrictions, or all of the above!) and the plugin will show you the ImproveOSM data as a separate layer. We also have a blog post about using the JOSM plugin.

Finally, a few interesting / funny examples of ImproveOSM data around the world.

ImproveOSM data points out that a new road alignment is now in use. Aerial imagery and OSM have not been updated yet. This is in northern Sweden.

Here, we stumble upon an undermapped town north of Surat, India. Of course, there are un- and undermapped areas everywhere in the world, but the ImproveOSM data shows that there are people driving around on these streets using a GPS enabled app or vehicle — people who would benefit from better OSM data in their everyday lives. It is not hard to find places like this around the world.

Finally, an animation showing clusters of ‘water’ tiles. This is a side effect of the partner data we process. Since it’s anonymized there is no way to say anything about why these traces exist. Useful for OSM? Perhaps.. Interesting? I think so!

Are you finding interesting, useful, funny or wrong data in ImproveOSM? Let us know! Happy Mapping!

Facebooktwitter

Is OpenStreetMap Big Data ready?

This article was written by Adrian Bona as a draft for a talk at State of the Map US in Boulder, Colorado this past month. The talk did not make it into the program, but the technology lives on as a central part of our OpenStreetMap technology stack here at Telenav. We will continue to deliver weekly Parquet files of OSM data. Adrian has recently moved on from Telenav, but our OSM team is looking forward to hearing from you about this topic! — Martijn

Getting started with OpenStreetMap at large scale (the entire planet) can be painful. A few years ago we were a bit intrigued to see people waiting hours or even days to get a piece of OSM imported in PostgreSQL on huge machines. But we said OK … this is not Big Data.Meanwhile, we started to work on various geo-spatial analyses involving technologies from a Big Data stack, where OSM was used and we were again intrigued as the regular way to handle the OSM data was to run osmosis over the huge PBF planet file and dump some CSV files for various scenarios. Even if this works, it’s sub-optimal, and so we wrote an OSM converter to a big data friendly columnar format called Parquet.The converter is available at github.com/adrianulbona/osm-parquetizer.Hopefully, this will make the valuable work of so many OSM contributors easily available for the Big Data world.

How fast?

Less than a minute for romania-latest.osm.pbf and ~3 hours (on a decent laptop with SSD) for the planet-latest.osm.pbf.

Getting started with Apache Spark and OpenStreetMap

The converter mentioned above takes one file and not only converts the data but also splits it in three files, one for each OSM entity type – each file basically represents a collection of structured data (a table). The schemas of the tables are the following:

node
 |-- id: long
 |-- version: integer
 |-- timestamp: long
 |-- changeset: long
 |-- uid: integer
 |-- user_sid: string
 |-- tags: array
 |    |-- element: struct
 |    |    |-- key: string
 |    |    |-- value: string
 |-- latitude: double
 |-- longitude: double

way
 |-- id: long
 |-- version: integer
 |-- timestamp: long
 |-- changeset: long
 |-- uid: integer
 |-- user_sid: string
 |-- tags: array
 |    |-- element: struct
 |    |    |-- key: string
 |    |    |-- value: string
 |-- nodes: array
 |    |-- element: struct
 |    |    |-- index: integer
 |    |    |-- nodeId: long

relation
 |-- id: long
 |-- version: integer
 |-- timestamp: long
 |-- changeset: long
 |-- uid: integer
 |-- user_sid: string
 |-- tags: array
 |    |-- element: struct
 |    |    |-- key: string
 |    |    |-- value: string
 |-- members: array
 |    |-- element: struct
 |    |    |-- id: long
 |    |    |-- role: string
 |    |    |-- type: string

Now, loading the data in Apache Spark becomes extremely convenient:

val nodeDF = sqlContext.read.parquet("romania-latest.osm.pbf.node.parquet")
nodeDF.createOrReplaceTempView("nodes")

val wayDF = sqlContext.read.parquet("romania-latest.osm.pbf.way.parquet")
wayDF.createOrReplaceTempView("ways")

val relationDF = sqlContext.read.parquet("romania-latest.osm.pbf.relation.parquet")
relationDF.createOrReplaceTempView("relations")


From this point on, the Spark world opens and we could either play around with DataFrames or use the beloved SQL that we all know. Lets consider the following task:

For the most active OSM contributors, highlight the distribution of their work over time.

The DataFrames API solution looks like:

val nodeDF = nodeDF
    .withColumn("created_at", ($"timestamp" / 1000).cast(TimestampType))
    .createOrReplaceTempView("nodes")

val top10Users = nodeDF.groupBy("user_sid")
    .agg(count($"id").as("node_count"))
    .orderBy($"node_count".desc)
    .limit(10)
    .collect
    .map({ case Row(user_sid: String, _) => user_sid })
    
nodeDF.filter($"user_sid".in(top10Users: _*))
    .groupBy($"user_sid", year($"created_at").as("year"))
    .agg(count("id").as("node_count"))
    .orderBy($"year")
    .registerTempTable("top10UsersOverTime")


The Spark SQL solution looks like:

select 
    user_sid, 
    year(created_at)) as year,
    count(*) as node_count
from 
    nodes
where 
    user_sid in (
        select user_sid from (
            select 
                user_sid, 
                count(*) as c 
            from 
                nodes 
            group by 
                user_sid 
            order by 
                c desc 
            limit 10
        )
    )
group by 
    user_sid, 
    year(created_at)
order by 
    year


Both solutions are equivalent, and give the following results:

alt tag

Even if we touched only a tiny piece of OSM, there is nothing to stop us from analyzing and getting valuable insights from it, in scalable way.

If you are curious about more advanced interaction between OpenStreetMap and Apache Spark, take a look at this databricks notebook.

OpenStreetMap Parquet files for the entire planet?

Telenav is happy to announce weekly releases of OpenStreetMap Parquet files for the entire planet at osm-data.skobbler.net.

Facebooktwitter

Find your MapRoulette Challenge

MapRoulette is a fun way to spend a few minutes (or hours…) improving OpenStreetMap. MapRoulette will present you with a random, easy to solve issue in OSM. MapRoulette is organized in ‘Challenges’, groups of tasks that are of the same nature. For example, there is a challenge to add missing crosswalks in various areas in Switzerland, based on analysis of aerial images.

How do you find a challenge you would like to work on? The MapRoulette home page provides a map of all the challenges, but this has some shortcomings. The challenge ‘centers’ are no

t always representative of where the tasks actually are located. It is also hard to search by topic. MapRoulette also has a search bar that you can use to find a challenge by keyword.

I want to work on making it much easier to find

 interesting MapRoulette challenges, and I would like to hear from you how you think that should work. Please add a comment below with your ideas!

In the mean time, I made a page that lists the most popular and newest challenges. It is a bit of a hack so let me know if it stops working 😉

Happy Mapping!

Facebooktwitter

Help fix up TIGER v1 ways

Old, untouched TIGER ways are still abundant in OSM 🙁 and fixing them up seems to be an endless task.

ugh!

I don’t know why I didn’t do this before, but I finally got around to making a MapRoulette challenge so we can fix them together:

>> Go to the challenge <<

Because the number of old TIGER ways is huge, this challenge covers only a tiny part of the U.S. as you can see here:

Once this part is done, we can reload the challenge with more old TIGER ways.

If you look at the screenshot above, you can also see what the query is that goes into Overpass to create the challenge in the first place. You can easily adapt it to make your own local challenge if you want to start fixing up old TIGER ways with your local mapping friends! (Why not organize a TIGER fixing party? OSM US will pay for pizza!)

If you’re interested in the Overpass details and some ideas for improving it, keep reading. Otherwise, just start fixing! 

Query Overpass for old TIGER

Here is my extremely simplified way to query Overpass for old TIGER ways:

way[highway]["tiger:tlid"](40, -113, 41, -111);
out body geom qt;

It takes the bounding box (40, -113, 41, -111) and searches for ways that have the highway tag as well as the tiger:tlid tag. This query should be a pretty good approximation of a real old TIGER way query, because the tiger:tlid tag is removed automatically when you edit such a way in iD or JOSM. So any way that still has this tag must not have been edited since the import.

This query falls short of a real old TIGER ways query, because the nodes that make up the way may very well have been edited. I am also not 100% sure under which circumstances the editors remove the tiger:tlid and other unnecessary TIGER import tags. It may be safer to look for last edited date or version number. If you have suggestions for improvement, please let me know in the comments.

Happy mapping!

Facebooktwitter

3D Scene reconstruction

From 2D images we can extract a limited range of information like width, height and color. These can be useful to determine the regions of interest in our images: street signs, lanes, or even roads.

However, for a more accurate detection, the depth perception
is crucial. Here comes the 3D reconstruction into play. Extracting a 3rd dimension, the depth, we can determine how far from the
camera the regions of interest are and consequently, their shape. This way we can distinguish the road from the obstacles (cars, pedestrians, curbstones) simply because we know that the road has an increasing distance from the camera while the objects have a constant distance (fig. 1).

fig. 1 Depth Map

Continue reading “3D Scene reconstruction”

Facebooktwitter

More and Updated Data for ImproveOSM

ImproveOSM has been updated with many new roads. We processed recent  GPS data from a number of data partners with some great results. A total of 30,000 new missing road tiles were added, over 17000 in Indonesia alone.

Aside from the missing roads, we added 67000 potential missing one-way roads that we detected with high confidence. Internal testing revealed only 6% false positives.

We are happy to continue providing OSM mappers with high quality data about missing things in OSM based on billions of GPS traces. Because ImproveOSM is based on actual drives from people using navigation or mapping software in their vehicles, and we apply a pretty high threshold for number of trips and quality of the GPS data, you can be pretty confident that every ImproveOSM feature will lead you to something you can add to OSM. Even if the aerial imagery is poor.

You should see the new data in your ImproveOSM plugin or on the ImproveOSM web site very shortly. Happy mapping and let us know what you mapped using ImproveOSM!

Facebooktwitter

New Version Of OpenStreetCam Introduces Points

Late last week, we released new versions of the OpenStreetCam apps and the web site. While we continue to make the platform faster and more reliable, we also like to keep adding interesting and fun features from time to time! This new release introduces points and levels. Every time you drive, you earn points. Earn enough points and you level up.

We went back in and calculated points for all your existing trips, so why not head to the newly designed leaderboard and see how you stack up against your fellow cammers? You can also see the leaderboard in the app:

We also enabled leaderboards by country on top of the daily, weekly and monthly rankings.

Your profile screen in the app and on the site will show you exactly how many points you have, how many you earned per trip, and what your current level is.

The new profile screen

More points for unexplored roads

So as you are driving around, you will automatically earn points for every picture recorded. But not all pictures earn you equal points! The less explored a road is, the more points you get — up to 10x the points for roads that have no coverage at all yet!

(This made it possible for me to gain 11k points on a 50 minute drive last week: most of the roads had no coverage yet, so I was getting 10x points for most of the way. )

11k points!

You can see which roads are less covered, or not covered at all yet, in the app. Just look for the roads with lighter or no purple OSC overlay:

Darker streets have better coverage, lighter streets need more.

We calculate the quality of coverage by the number of trips that cover the way as well as the age of the existing trips. This way we encourage each other to always have the most recent imagery available for OpenStreetMap.

We hope you enjoy the new features! Please let us know what you think by writing us at hello@openstreetcam.org.

Facebooktwitter

OSMTime in Cluj featuring MapRoulette

OSMTime is a monthly OSM mapping event organized by Telenav colleague Beata Jancso. Telenav hosts the events in the Cluj-Napoca office  and sponsors with pizza. Usually Bea chooses a theme and sometimes there will also be a speaker with an interesting OSM related topic.

While visiting the Telenav Romania office in Cluj last week, I was lucky to also catch an OSMTime event. The theme of the evening was ‘Mapping Roundabouts using MapRoulette’. Being the person behind MapRoulette, Bea asked me to do a quick introduction. Colleague Bogdan Gliga also presented the metodology he used to detect missing roundabouts from massive amounts of probe data. (He wrote about that topic here as well.)

OSMTime Cluj with Bogdan presenting
OSMTime Cluj with Bogdan presenting

After the presentations and pizza, the 25 or so mappers logged on to MapRoulette to start with the new Missing Roundabouts challenge. Most people had not used MapRoulette before, so I was glad that everyone was getting the hang of it quickly. Most of the problems and questions were not about MapRoulette but about what is a roundabout exactly, and what is the difference between a roundabout and a mini_roundabout and a traffic_circle. (The OSM wiki helps out a little here.)

At the end of the evening, the mappers in the room already made a good dent in the challenge, which has more than 4500 tasks total.

I had a great time, thanks to Bea for organizing the OSMTime events every month and spreading the word. If you are in the Cluj-Napoca area, you may want to subscribe to the OSMTime meetup so you know when the next one takes place. Or look for an OpenStreetMap meetup in your area and meet local mappers!

Facebooktwitter

Usage tips on the new Improve OSM

In this post we will take a detailed look at the new improveosm.org web site. We recently completely overhauled the application. It is now based on the OSM iD editor. We will walk you through the functionality and give some pointers.

Zoom and layer activation

Just by entering the web application, you already have our Improve OSM layer selected, and also active. A low zoom level displays a heat-map the content of which is modifiable via the left side filtering options. Note the color coded dots, corresponding to the heat-map’s circles.

improve-osm-heatmap

A higher zoom level, meaning a zoom level over 15, will show each individual Improve OSM item. Turn Restrictions are grouped in clusters, if some items represent rules for the exact same intersection. You’ll also notice that different types of Missing Roads have their specific color on the map and are also marked with the respective color in the filter panel. Filtering has the same usage as in the heat-map.improve-osm-transparent-tr

We talked about the Improve OSM layer being active by default, but what exactly does that mean? It means you can use the filters from the side panel and you can also interact with any Improve OSM item on the map. While inactive, the filter panel becomes minimized and any Improve OSM item becomes see-through. You can’t interact with the layer in this inactive state, but you can and will want to interact with all the iD’s map items and editing mechanisms. The SPACE key switches between the active and the inactive states. You can also use the toggle button found in the side panel’s header.

Item selection and status

When the Improve OSM layer is active any item can be selected. After selection, options appear in the side panel allowing you to change the status and add a comment.

You can have a multiple selection by keeping the CTRL key pressed and selecting items with the mouse. Only for the Missing Road items, you can batch select neighboring tiles. For this, keep the SHIFT key pressed and click one of the tiles in the batch. All neighboring tiles in that batch will be selected.

improve-osm-selections-1

improve-osm-selections-2

For Turn Restriction clusters, selecting one will automatically select a single Turn Restriction from the cluster. You can switch to another item in that cluster by selecting it from the list appearing in the side panel.

improve-osm-selections-3

Clicking anywhere on the map area not containing an Improve OSM item, will deselect all selected items.

For Turn Restrictions, you can select one to see its describing arrows and pass numbers, and then have the entire Improve OSM layer inactivated. This way, you’ll still see the describing arrows while being able to interact with the editing iD tools.

Item types

Turn Restrictions – they mark an intersection where a new, unmapped, turn restriction is in place. The green arrow marks the street vehicles came from while entering an intersection, while the red one marks a street used by none or very few of those vehicles to exit the intersection. It’s assumed that the very low percent of passes on the red segment, indicates a strong possibility of a turn restriction being in place. Passes on the red segment are assumed to be traffic rules violations.

Missing Roads – they mark a portion of the map (a tile) that contains GPS output from vehicles, in places no road is mapped. If they are numerous and they represent a clear trail, it’s assumed a missing road is there.

One Ways – they mark a road that is not mapped as a one way street, for which traffic data suggests with high confidence it actually is a one way street.

A typical usage flow

1. Identify an Improve OSM marker (item) and decide if iD editing is required

improve-osm-flow-set-1

2. Press SPACE to inactivate the Improve OSM layer and use the iD’s tools

improve-osm-flow-set-2

3. Make the wanted edit in iD

improve-osm-flow-set-3

4. Press SPACE agai, select the resolved Improve OSM item and change the item’s status to ‘SOLVED’

improve-osm-flow-set-4

We hope you find the new improveosm.org web site useful and are looking forward to hearing your feedback!

Facebooktwitter