Use PGRouting to handle routing directly in Database through API #4070

justinefricou · 2024-04-15T09:22:20Z

Description

Move lienar routing from frontend to backend
Use PGRouting Djikstra algorythm

Related Issue

Checklist

geotrek/core/static/core/multipath.js

geotrek/core/views.py

Resolved

geotrek/core/views.py

geotrek/core/tests/test_path_router.py

…ndling whole edges with their default direction)

…es with their default direction)

…include more cases

…ible

…ute modification

* Add TrekGeometry view with get and post methods * Add a parameter to TrekGeometry post method * Add graph and steps info being sent to TrekGeometry view * Add computing of the dijkstra matrix and start and end nodes (only handling whole edges with their default direction) * Add computing of the path to follow (node to node) (handles whole edges with their default direction) * Modify path in response: it's now from start to end * Refacto: nodes idx to id correspondence * Add structure of algorithm handling partial edges and more than 2 steps * ADD handling of edge percentage and more than 2 markers * Add display of a hard-coded polyline sent from the TrekGeometry view * The graph is directly taken from the backend * Add handling of GPS coordinates as input * Add use of Point ewkt when using sql * Add TODOs for next step: obtaining a LineString from dijkstra results * Starting to work on converting the resulting nodes list into LineStrings * Add conversion of dijstra results into a displayed Linestring * Correctng some comments * Fix path not computed if adjacent steps are on the same edge * Fix negative dijkstra weight when we're going backwards relative to a Path direction * Fix path not computed when going backwards on a Path if going through several edges * Starting to display fetched route instead of locally computed route * Preparing topology_helper to handle several linestrings * Automating the geojson generation in the view * Add handling of several steps * Replacing MultiLineString by GeometryCollection in Response * Fixing the organization of layers in the displayed path * Removing old code related to the graph * Removing old code * Removing old code * Making the TrekGeometry post method into a PathViewSet action * Reordering PathRouter methods * Removing graph action of PathViewSet * Moving initial graph generation into the PathRouter class * Removing no longer used imports * Renaming graph.py to path_router.py * Adding parameters error handling for the route_geometry action * Using dict.update instead of a custom method * Adding status codes to Responses * Optimizing the cs_graph generation (generating the upper triangle then filling the lower triangle symmetrically) * Optimizing matrix generation: it is now generated at init and modified depending on the current steps * Optimizing matrix generation by using its symmetrical property * Refactoring: making duplicate code into a method * The route is now fetched only when the marker dragging stops * Saving the dijkstra matrix in the cache * Saving the path graph in the cache * Moving duplicate cache-related code into a single method * Add handling of an impossible path: the view returns a status 204 and the js does not attempt to display the path * Modify error handling in js: using Promise.reject * Correct returns when no path could be found Methods that return an array when a path is found now return [] instead of None if no path can be found * Replaces status 204 by status 400 when no path can be found * Add a spinner when the route is being fetched * Add sending of the modified steps indexes to fetchRoute * Adds fetching of the route for only the modified steps * Add algorithm that updates the displayed route layers * Add use of currently displayed layers indexes to replace them * Fix replacing the currently displayed layers indexes by new ones * Fix updating layers not working after removing a marker * Bump leaflet.textpath.js to v1.1.0 * Move leaflet static files into a vendor directory * Bump leaflet.lineextremities.js to v0.1.1 * Fix missing path layer when an unlinkable via marker is removed * Remove console logs * Add coloring of the markers when one is isolated * Add coloring of all markers to normal appearance when back to normal * Add enabling/disabling of markers drag when a route is correct/incorrect * Add enabling/disabling of markers deletion when a route is correct/incorrect * Add impossibility of creating new via-steps while the route is invalid * Move spinner.stop() into a finally() * Fix one route layer missing when an isolated via-step is corrected * Add deletion of a marker if not dropped on a path * Add resetting of a marker to its previous valid position when not dropped on a path * Fix isolated marker highlight being removed when reset to its previous position * Remove duplicate part of code * Remove hiding of route layer when the start or end marker is moved * Fix out of date step idx used when deleting a via point * Add display of an error toast when a marker is incorrect * Modify error_toast elements class names for genericity * Fix toast stacking: top toast not on the bottom when it's the only one displayed * Fix wrong marker being deleted after trying to create an unsnapped marker * Fix route layer not displayed after start and end markers created and unlinkable * Add disabling of markers when the route is being fetched * Add saving of the route topology through the form * Add modification of the route topology through the edit form * Fix Geotrek object no longer created before use * Remove use of non-sparse matrix for dijkstra (using csr_array) * Modify use of sparse array (using lil_array instead of csr_array) * Optimize dijkstra matrix generation * Remove calls to now deleted js scripts * Remove no longer used methods * Add benchmarking system based on frontend routing version * Fix typo in cypress.config.js * Startng to work on adapting frontend benchmark system for backend * Add record of the number of runs for a benchmark measure * Modify route control clicking via cypress - Path elements' data-test attributes are now set when they are added to the page - Cypress test waits for the data-test attribute to be added before clicking on the route control * Replace broken 3 via-pts topology with a 2 via-pts topology * Modify cypress benchmark: adapt the frontend version to work with backend * Modify python time measures: now in milliseconds instead of seconds * Modify time measurement: startTime is now recorded before the click/drop event * Modify benchmark.sh: making the session id a variable * Add scenario using the big DB and no via point * Add display of current branch name for each measure * Replace big database no via pts scenario with 25 via pts scenario * Add readme for the benchmarking system * Modify benchmarking script: cypress spec file and session id are now passed as command line arguments * Fix crash when computing time averages if a file is missing * Add 2 via-pts topology for big database * Replace 25-via-pts scenario with 2-via-pts scenario for big database * Add unit tests for route_geometry view (check of request body) * Modify cypress timeouts: now 30min * Start to work on impossible and possible path test cases * Add check of lat and lng type and range in route_geometry view * Fix request body not sent in the right format (route_geometry tests) * Add new scenario for big DB with no via point * Add test case: steps not on paths * Add test case: steps one paths not linked * Add tests for route_geometry view * Fix test for the route_geometry view * Remove prints * Move benchmarking directory into tools/ * Fix norm * Modify Cypress 'create trek' test after routing changes * Try to fix cypress e2e tests * Uncomment env_dev settings * Modify Cypress e2e tests: actually click on paths to create a trek * Remove console.log and debugger * Fix cypress e2e tests: id of clicked path * Add test case for route_geometry view: error 500 * Try to fix cypress tests: ensure the topology control is enabled before clicking on it * Comment content of test_graph temporarily * Adapt graph generation tests to the new routing system * Add tests for path_router * Add unit test for cache handling in path_router * Fix cache-related test for PathRouter * Modify test_route_geometry_not_fail_with_via_points_several_paths to include more cases * Modify route_geometry tests: more precision when checking results * Remove old unit test for route_geometry view * Remove redundant unit test for path_router * Add script to take backend time measures * Add steps for route with 100 via-pts for backend time measures script * Fix topology used in 100-via-pts fixture * Add big database topology in backend time measuring script * Improve readability of path_router test: using yied * Update benchmark README to include the new script * Fix requirements.txt after rebase * Modify leaflet map: start spinner and disable markers ASAP * Modify leaflet map: stop spinner and reenable markers as late as possible * Adapt backend benchmark script to new PR * Modify display of route layers: only redisplay those affected by a route modification * Fix route modif not saved if an incorrect marker has been fixed * Fix path layers not displayed at creation * Add computing of the dijkstra matrix and start and end nodes (only handling whole edges with their default direction) * Modify docker-compose to use pgrouting image * Add routing with pgRouting for hard-coded topology * Add use of request latlngs for pgRouting * Generate linestrings for routes with no via-pts * Add display of the route (geometry is wrong on last edge) * Add generation of the serialized topology * Remove old dijkstra-related methods * Remove old dijkstra-related imports * Fix route always containing the whole last path instead of a fraction of it * Replace call to ST_Linesubstring by ST_SmartLineSubstring to split start and end edges * Fix route geometry reversed when a path is taken from its end to its start * Remove LineString rounding before merging * Fix route topology not correct when a path is taken in reverse * Remove prints * Uncomment try except * Add source and target columns to core_path table for pgRouting * Add error handling when no path is found * Remove out of date unit tests * Add use of settings PATH_SNAPPING_DISTANCE for pgRouting tolerance * Add extensions required for database * Modify database extensions * Remove unused imports after rebase * Fix postgres images for github CI * Add setting of source and target to null for a path that is being split * Modify routing: don't try to route on draft paths * Fix routing taking too much time (unnecessary recalculation of graph) * Move setting of pgRouting-related values to null from specific case to trigger after geom is updated * Fix PathViewsTest test_route_geometry_not_fail_no_via_point_one_path * Fix no route found when a marker is on a path's extremity (0 or 1) * Fix verification of serialized positions in tests (wrong format) * Modify how the closest path is found: get it from the front rather than the back * Fix routing tests: create a path before sending the request with a path_id * Fix norm * Delete now empty test_path_router.py * Modify routing conditions: don't route on paths whose visiblity is set to false * Remove modification of geom_3d as a condition for setting pgRouting-related values to null in trigger * Fix incorrect path id test (wrong lng used) * Add checking of error messages for route_geometry unit tests * Add more detailed check of response values for route_geometry tests * Add test cases for route_geometry: draft and invisible paths, editing geom, adding / deleting paths * Add django command to generate pgRouting network topology * Add unit test for command generate_pgr_network_topology * Add entry in doc explaining how to use the generate_pgr_network_topology command * Move tests docstrings descriptions to top * Add docstrings to path_router * Adapt backend benchmark script for new PR * manage pgrouting in nstall scripts * enable pgrouting extension in debian postinst * Update benchmark system for pgRouting * fix tests and viewset permission * prevent flaky test * Update docs/install/import.rst * fix rebase * fix rebase --------- Co-authored-by: JustineFricou <[email protected]> Co-authored-by: J-E Castagnede <[email protected]>

submarcos · 2024-09-03T09:29:43Z

geotrek/core/tests/test_path_router.py

+        }
+
+        graph = self.path_router.graph_edges_nodes_of_qs(Path.objects.order_by('id'))
+        self.assertDictEqual(graph, graph)


Graph is always equals to graph.

You should use another variable name and compare both

submarcos · 2024-09-13T11:50:12Z

geotrek/core/templates/core/error_toast.html

@@ -0,0 +1,11 @@
+<div id="{{id}}" class="toast toast-error text-white" role="alert" aria-live="assertive" aria-atomic="true">


Suggested change

<div id="{{id}}" class="toast toast-error text-white" role="alert" aria-live="assertive" aria-atomic="true">

<div id="{{ id }}" class="toast toast-error text-white" role="alert" aria-live="assertive" aria-atomic="true">

submarcos · 2024-09-13T11:50:25Z

geotrek/core/templates/core/error_toast.html

@@ -0,0 +1,11 @@
+<div id="{{id}}" class="toast toast-error text-white" role="alert" aria-live="assertive" aria-atomic="true">
+  <div class="toast-header toast-error-header">
+    <strong class="mr-auto">{{title}}</strong>


Suggested change

<strong class="mr-auto">{{title}}</strong>

<strong class="mr-auto">{{ title }}</strong>

submarcos · 2024-09-13T11:50:38Z

geotrek/core/templates/core/error_toast.html

+    </button>
+  </div>
+  <div class="toast-body toast-error-body">
+    <span>{{message}}</span>


Suggested change

<span>{{message}}</span>

<span>{{ message }}</span>

submarcos · 2024-09-13T11:52:54Z

requirements.txt

@@ -240,6 +240,7 @@ numpy==1.23.4
    # via
    #   large-image
    #   large-image-source-vips
+    #   scipy


Suggested change

# scipy

requirements.txt

delete scipy dependencies

This comment was marked as outdated.

Sign in to view

Chatewgne reviewed Apr 19, 2024

View reviewed changes

geotrek/core/static/core/multipath.js Outdated Show resolved Hide resolved

justinefricou force-pushed the move_routing_to_backend branch from 7e1392b to d793ab7 Compare April 26, 2024 09:38

Chatewgne previously requested changes Apr 29, 2024

View reviewed changes

geotrek/core/views.py Show resolved Hide resolved

geotrek/core/views.py Outdated Show resolved Hide resolved

geotrek/core/views.py Outdated Show resolved Hide resolved

justinefricou force-pushed the move_routing_to_backend branch from 1e85200 to 338c6a1 Compare May 3, 2024 09:40

justinefricou force-pushed the move_routing_to_backend branch from e3bdca4 to 685a790 Compare May 17, 2024 13:10

justinefricou force-pushed the move_routing_to_backend branch 2 times, most recently from c9952df to 3bc7f0e Compare June 4, 2024 14:45

justinefricou force-pushed the move_routing_to_backend branch 2 times, most recently from 561a44b to dab9c50 Compare June 12, 2024 15:59

justinefricou force-pushed the move_routing_to_backend branch 3 times, most recently from 8258a95 to 10425c8 Compare July 9, 2024 14:14

justinefricou force-pushed the move_routing_to_backend branch from 4f94203 to a4d732b Compare July 11, 2024 08:47

Chatewgne reviewed Jul 11, 2024

View reviewed changes

geotrek/core/views.py Show resolved Hide resolved

geotrek/core/views.py Show resolved Hide resolved

geotrek/core/views.py Outdated Show resolved Hide resolved

geotrek/core/tests/test_path_router.py Outdated Show resolved Hide resolved

justinefricou force-pushed the move_routing_to_backend branch 2 times, most recently from a843b59 to e1311b9 Compare July 30, 2024 08:24

justinefricou force-pushed the move_routing_to_backend branch from 7c0b5ba to 42e23bc Compare August 28, 2024 07:35

justinefricou mentioned this pull request Aug 30, 2024

Optimisation du système de routage #4286

Open

JustineFricou added 9 commits September 4, 2024 13:05

Add TrekGeometry view with get and post methods

8c79d9e

Add a parameter to TrekGeometry post method

4744348

Add graph and steps info being sent to TrekGeometry view

a290ade

Add computing of the dijkstra matrix and start and end nodes (only ha…

9735876

…ndling whole edges with their default direction)

Add computing of the path to follow (node to node) (handles whole edg…

7263e8e

…es with their default direction)

Modify path in response: it's now from start to end

d0e1423

Refacto: nodes idx to id correspondence

3af92b2

Add structure of algorithm handling partial edges and more than 2 steps

cd043bd

ADD handling of edge percentage and more than 2 markers

a28c00c

justinefricou added 18 commits September 4, 2024 13:05

Fix cache-related test for PathRouter

86c7725

Modify test_route_geometry_not_fail_with_via_points_several_paths to …

fa8c44d

…include more cases

Modify route_geometry tests: more precision when checking results

a15c53f

Remove old unit test for route_geometry view

3da50d0

Remove redundant unit test for path_router

74c72f3

Add script to take backend time measures

f836d06

Add steps for route with 100 via-pts for backend time measures script

7f3fd07

Fix topology used in 100-via-pts fixture

8cfd90a

Add big database topology in backend time measuring script

553f361

Improve readability of path_router test: using yied

0201aeb

Update benchmark README to include the new script

2efd83b

Fix requirements.txt after rebase

0530be6

Modify leaflet map: start spinner and disable markers ASAP

3eadf2b

Modify leaflet map: stop spinner and reenable markers as late as poss…

de17dee

…ible

Adapt backend benchmark script to new PR

488ffb2

Modify display of route layers: only redisplay those affected by a ro…

ad2d965

…ute modification

Fix route modif not saved if an incorrect marker has been fixed

58924cd

Fix path layers not displayed at creation

7e9c1eb

submarcos force-pushed the move_routing_to_backend branch from 42e23bc to 7e9c1eb Compare September 4, 2024 11:05

submarcos changed the title ~~Move routing to backend~~ Use PGRouting to handle routing directly in Database through API Sep 13, 2024

submarcos added ♻️ Refactoring 💫 Improvements ⚡ Performances labels Sep 13, 2024

submarcos reviewed Sep 13, 2024

View reviewed changes

submarcos and others added 5 commits September 16, 2024 10:43

Apply suggestions from code review

476e394

delete scipy dependencies

🐛 [BUG] Fix reset layer

a3a1a4c

🏗 [MAINT] Remove unused dependency scipy

acb8d7a

🏗 [MAINT] Remove unused dependency scipy

6729f20

Merge branch 'master' into move_routing_to_backend

4a85772

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use PGRouting to handle routing directly in Database through API #4070

Use PGRouting to handle routing directly in Database through API #4070

justinefricou commented Apr 15, 2024 •

edited by submarcos

Loading

This comment was marked as outdated.

This comment was marked as outdated.

submarcos Sep 3, 2024

submarcos Sep 13, 2024

submarcos Sep 13, 2024

submarcos Sep 13, 2024

submarcos Sep 13, 2024

		@@ -0,0 +1,11 @@
		<div id="{{id}}" class="toast toast-error text-white" role="alert" aria-live="assertive" aria-atomic="true">

	<div id="{{id}}" class="toast toast-error text-white" role="alert" aria-live="assertive" aria-atomic="true">
	<div id="{{ id }}" class="toast toast-error text-white" role="alert" aria-live="assertive" aria-atomic="true">

	<strong class="mr-auto">{{title}}</strong>
	<strong class="mr-auto">{{ title }}</strong>

Use PGRouting to handle routing directly in Database through API #4070

Are you sure you want to change the base?

Use PGRouting to handle routing directly in Database through API #4070

Conversation

justinefricou commented Apr 15, 2024 • edited by submarcos Loading

Description

Related Issue

Checklist

This comment was marked as outdated.

This comment was marked as outdated.

submarcos Sep 3, 2024

Choose a reason for hiding this comment

submarcos Sep 13, 2024

Choose a reason for hiding this comment

submarcos Sep 13, 2024

Choose a reason for hiding this comment

submarcos Sep 13, 2024

Choose a reason for hiding this comment

submarcos Sep 13, 2024

Choose a reason for hiding this comment

justinefricou commented Apr 15, 2024 •

edited by submarcos

Loading