Announcing The New Trailforks Route Planner

We are excited to announce the beta of the Trailforks Route Planner v2. I first created our route planner that allowed automatic A to B routing along trails back in 2015. It was updated along the way to include OSM roads nearby trails, a "selecting" method and last year a freehand drawing option with snapping to trails. But the original auto-routing was aging and needed to be updated to something faster and more flexible. Since we created the trail route planner in 2015 other websites created their own route planners, so I wanted to create the best and most flexible route planner for biking, hiking and other outdoor sports.

Our backend routing system and data pipeline has been totally re-worked using a new tile based routing server called Valhalla. I'll get more into the technical details near the bottom of the article. What this means to the end user is automatic routing is MUCH faster, supports more OSM roads and larger geographic scope. But that is just the start of the changes. I also re-made the front-end interface of the route planner and allow creating a route with a mixture of auto-routing, freehand drawing, uploading a GPX and manually editing GPS points. The old route planner you had to choose and stick to one. The new route planner also allows you to create an entire custom map for your route with POIs, polygons, lines and colored sections. I also generate much better turn-by-turn directions, surface type breakdown, snapping of messy GPS traces to the trails and more!

The Trailforks route planner is unique compared to other route creation tools in that it's not solely based on OSM data and allows you to create a route anywhere on the map quickly.

Here is a quick video demo of some of the new functionality.




More Feature Details



Create your perfect route with a combination of clicking points to route between them, dragging the last blue routing point to free-hand draw anywhere, or edit the raw GPS points for complete fine control.































Why The Change?

"I liked the old route planner, why did you change it!?" While the new route planner has A LOT of new functionality. The old route planner was often broken in various regions around the world, and the route planner would be a blank map. It also used an old Leaflet map missing many mapping features of the other newer maps on Trailforks. We don't want to maintain the old Leaflet based map or the old routing network system that was prone to break and be outdated. The new system uses the same Mapbox powered map used elsewhere on Trailforks with all the same features. The old planner was very limited because you could not mix or change types of routing, you had to stick to one, which was a very frustrating experience if there was some missing routing data or connection. The new system also includes MUCH more OSM roads that are routable and is a much faster at calculating auto-routing.



Nerd Talk, Tech Details


The old routing backend used pgRouting and was complex and slow to update. The routable areas were small, no larger than half a state, or sometimes only a single riding area. One could not route between them and it was kind of slow. Updating these routing areas was very slow, often taking several hours for a single large routing area. I've scrapped all that and moved to using a tile based routing server called Valhalla. It is MUCH faster at creating the route and allowed me to create a much better backend data pipeline that is faster to update and allows larger routable areas.

Similar to the old pgRouting I first need to turn the Trailforks data into connected "edges". Which means snapping all the trail intersections together and separating trails at intersections into separate tracks. I also need to snap the trails to OpenStreetMap roads we will include in the routing. This process can take a long time, so I've moved some of this to happen each time a trail is added or edited to Trailforks. Trails snap together and snap to OSM roads all the time now, which also makes a cleaner looking map and reduces the work needed for updating the routing data.

The old system had many issues with updating the routing data for an area, when a trail was added or edited, the area would be put in a queue to be updated, but often would take days or would fail and never update, or could break and no routing in that area would work. The new system no longer has a queue and the entire world will be updated at once every 2 days (might increase to daily). So when a new trail is added, it could take up to 2 days before it can be auto-routable, but you can always free-hand draw along it! Additionally if for some reason the routing server is offline, you can still see the trails on the route planner and create a route by free-hand drawing and snapping to the trails.

This world-wide update process involves separating the world into 27 routing areas, much larger than the previous routing areas, which there were many thousands. I do this to reduce the time it takes to generate the routing data and keep memory usage under control. I first make sure all the trails are snapped and segmented into edges for a routing area, then the data is combined with OSM roads and written to .OSM files. Since we are having to update our routing often, we are not including ALL OSM data everywhere, only roads within 10km of a trail are included. If we including ALL OSM roads, it could easily take 24 hours to update and could run into memory bottlenecks. These 27 generated OSM files are then merged into a single PBF file which Valhalla converts into routing tiles. What was particularly tricky was the OSM data needed to have all unique ids in sequential order, so this took a lot of memory to have all the nodes in memory to reference each other and sort. I spent a lot of time trying to speed up this process and have it use less memory. Currently the whole process to update the entire worlds routing takes around 5 hours, takes 242 GB of total memory and creates over 42 million routable edges with 236 million nodes. We have some serious server power behind this to make it fast and offer redundancy.

This page https://www.trailforks.com/routing_status/ lists all the routing areas, their data and the last time they were updated.



Issues

While we try to maintain good trail data quality through our local admins, mods and ambassadors. There are tens of thousands of trails being added and edited each month and some lower quality data slips through, or trails are added that are not connected to other trails and roads. We have automated checks to try forcing the data to "snap" to nearby trails and roads, but sometimes the data is outside the tolerance. We also have checks that flag trail data with quality issues, for admins to cleanup and improve over time. But there can still be missing or broken trail connections in the routing network that breaks the "auto-routing". Over time users and admins will fix and improve the trail data making a better connected routing network. But the big advantage of the new route planner is you can bypass any routing issue but freehand drawing a section of the route, or holding ALT and drawing a straight line over the missing connection.

Admins have debug options to view the routing network data (in pink).


The new route planner is still being developed and improved. There are some quirks currently, like not being able to remove auto-routing points after you mix in freehand drawing. This also limits us from allowing editing auto-routing points after being placed, this is something we'd like to improve.

Try out the Trailforks Route Planner.