Scans interrupted at high percentage

[cpbruno]

Expected behavior

GVM to complete the scan (reach 100% change its status from Running to Complete and Run the Alerts).

Actual behavior

GVM stuck at 99%. It has happened at 80%, and another few times at 95%+.

Steps to reproduce

This is hard as I couldn't always reproduce. At some Target lists, it completed flawlessly and in a fair amount of time. On other Targets, it can take up to 3 days stuck at 99% when it switches to Interrupted.

I was not sure what was causing it until find out recent logs and maybe this can help to solve it? On the last 4 scans that got stuck at 99% I saw the same error: ERROR: canceling statement due to user request

GVM versions

gsa: (gsad --version)

GSAD version 22.04.0

gvm: (gvmd --version)

Greenbone Vulnerability Manager version 22.4.0~dev1 (DB revision 250)

openvas-scanner: (openvas --version)

OSPd OpenVAS version 22.4.3.dev1.

GSAD version 22.04.0

gvm-libs:

latest at the moment of writing:
Digest:sha256:95f6eee25df254ede2a60ef0fdb8ed1e6a663835eaee16808c1bcd1a96ab7f87

Environment

Operating system:

Shouldn't matter but:

Distributor ID: Ubuntu
Description:    Ubuntu 22.04.1 LTS
Release:        22.04
Codename:       jammy

Installation method / source:

Greenbone GVM docker images with community docker-compose services.

Logfiles

gsa_1                  | gsad  gmp:WARNING:2022-11-25 16h38.13 GMT:13: Authentication failure for 'gvm' from 192.168.18.9. Status was 2.                                                                                                      
gvmd_1                 | md    gmp:WARNING:2022-11-25 16h38.13 utc:6547: Authentication failure for 'gvm' from unix_socket                                                                                                               
gsa_1                  | gsad  gmp:MESSAGE:2022-11-25 16h38.41 GMT:13: Authentication success for 'admin' from 192.168.18.9                                                                                                   
gsa_1                  | libgvm util:WARNING:2022-11-25 16h38.45 GMT:13:    End error: Error on line 1 char 1: Document was empty or contained only whitespace                                                                                
pg-gvm_1               | 2022-11-25 16:38:46.728 UTC [4157] gvmd@gvmd ERROR:  canceling statement due to user request                                                                                                                         
pg-gvm_1               | 2022-11-25 16:38:46.728 UTC [4157] gvmd@gvmd CONTEXT:  SQL function "report_severity_count" statement 1                                                                                                              
pg-gvm_1               | 2022-11-25 16:38:46.728 UTC [4157] gvmd@gvmd STATEMENT:  SELECT sum(aggregate_count) AS outer_count, aggregate_group_value AS outer_group_column, aggregate_subgroup_value AS outer_subgroup_column , max (text_column_0), max (text_column_1), max (text_column_2), max (text_column_3) FROM (WITH permissions_subject     AS (SELECT * FROM permissions         WHERE subject_location               = 0         AND ((subject_type = 'user'               AND subject                   = (SELECT id FROM users WHERE users.uuid = '933ce1ed-06f4-471e-aa20-debfd2bd76d6'))              OR (subject_type = 'group'                  AND subject                      IN (SELECT DISTINCT "group"                          FROM group_users                          WHERE "user"                                = (SELECT id FROM users WHERE users.uuid = '933ce1ed-06f4-471e-aa20-debfd2bd76d6')))              OR (subject_type = 'role'         AND subject                      IN (SELECT DISTINCT role                          FROM role_users                          WHERE "user"                                = (SELECT id FROM users WHERE users.uuid = '933ce1ed-06f4-471e-aa20-debfd2bd76d6'))))),     super_on_users     AS (SELECT DISTINCT *         FROM (SELECT resource FROM permissions_subject               WHERE name = 'Super'               AND resource_type = 'user'               UNION               SELECT "user" FROM role_users               WHERE role                     IN (SELECT resource                         FROM permissions_subject                         WHERE name = 'Super'                         AND resource_type = 'role')               UNION               SELECT "user" FROM group_users               WHERE "group"                     IN (SELECT resource                         FROM permissions_subject                         WHERE name = 'Super'        AND resource_type = 'group'))              AS all_users)SELECT  count(*) AS aggregate_count, uuid AS aggregate_group_value, CAST (NULL AS TEXT) AS aggregate_subgroup_value, max (CASE WHEN target IS null OR opts.ignore_severity != 0 THEN 0 ELSE coalesce (report_severity_count (task_last_report (id),                                 opts.override, opts.min_qod,                                 'High') * 1.0            / nullif (report_result_host_count (task_last_report (id),                                                opts.min_qod), 0),          0) END) as text_column_0, max (modification_time) as text_column_1, max (name) as text_column_2, max (task_severity (id, opts.override, opts.min_qod)) as text_column_3 FROM tasks, (SELECT   0 AS override,   70 AS min_qod,   0 AS ignore_severity)  AS opts WHERE  ((tasks.owner   = (SELECT id FROM users WHERE users.uuid = '933ce1ed-06f4-471e-aa20-debfd2bd76d6'))  OR EXISTS (SELECT * FROM permissions_subject             WHERE name = 'Super'             AND (resource = 0))  OR tasks.owner IN (SELECT *                     FROM super_on_users)  OR EXISTS (SELECT id FROM permissions_subject  WHERE resource = tasks.id  AND resource_type = 'task'  AND resource_location = 0  AND (t ()))) AND hidden = 0 AND usage_type = 'scan'   GROUP BY uuid  )      AS agg_sub GROUP BY outer_group_column, outer_subgroup_column ORDER BY outer_group_column DESC, max (text_column_0) DESC LIMIT 10 OFFSET 0;
gsa_1                  | libgvm util:WARNING:2022-11-25 16h38.45 GMT:13:    End error: Error on line 1 char 1: Document was empty or contained only whitespace
pg-gvm_1               | 2022-11-25 16:38:46.956 UTC [4155] gvmd@gvmd ERROR:  canceling statement due to user request 
pg-gvm_1               | 2022-11-25 16:38:46.956 UTC [4155] gvmd@gvmd CONTEXT:  SQL function "report_severity_count" statement 1                                                                  
pg-gvm_1               | 2022-11-25 16:38:46.956 UTC [4155] gvmd@gvmd STATEMENT:  SELECT sum(aggregate_count) AS outer_count, aggregate_group_value AS outer_group_column, aggregate_subgroup_value AS outer_subgroup_column , max (text_column_0), max (text_column_1), max (text_column_2), max (text_column_3) FROM (WITH permissions_subject     AS (SELECT * FROM permissions         WHERE subject_location               = 0         AND ((subject_type = 'user'               AND subject                   = (SELECT id FROM users WHERE users.uuid = '933ce1ed-06f4-471e-aa20-debfd2bd76d6'))              OR (subject_type = 'group'                  AND subject                      IN (SELECT DISTINCT "group"                          FROM group_users                          WHERE "user"                                = (SELECT id FROM users WHERE users.uuid = '933ce1ed-06f4-471e-aa20-debfd2bd76d6')))              OR (subject_type = 'role'      AND subject                      IN (SELECT DISTINCT role                          FROM role_users                          WHERE "user"                                = (SELECT id FROM users WHERE users.uuid = '933ce1ed-06f4-471e-aa20-debfd2bd76d6'))))),     super_on_users     AS (SELECT DISTINCT *         FROM (SELECT resource FROM permissions_subject               WHERE name = 'Super'               AND resource_type = 'user'               UNION               SELECT "user" FROM role_users               WHERE role                     IN (SELECT resource                         FROM permissions_subject                         WHERE name = 'Super'                         AND resource_type = 'role')              UNION               SELECT "user" FROM group_users               WHERE "group"                     IN (SELECT resource                         FROM permissions_subject                         WHERE name = 'Super'                    AND resource_type = 'group'))              AS all_users)SELECT  count(*) AS aggregate_count, uuid AS aggregate_group_value, CAST (NULL AS TEXT) AS aggregate_subgroup_value, max (CASE WHEN target IS null OR opts.ignore_severity != 0 THEN 0 ELSE coalesce (report_severity_count (task_last_report (id),                                 opts.override, opts.min_qod,                                 'High') * 1.0            / nullif (report_result_host_count (task_last_report (id),                                                opts.min_qod), 0),          0) END) as text_column_0, max (modification_time) as text_column_1, max (name) as text_column_2, max (task_severity (id, opts.override, opts.min_qod)) as text_column_3 FROM tasks, (SELECT   0 AS override,   70 AS min_qod,   0 AS ignore_severity)  AS opts WHERE  ((tasks.owner   = (SELECT id FROM users WHERE users.uuid = '933ce1ed-06f4-471e-aa20-debfd2bd76d6'))  OR EXISTS (SELECT * FROM permissions_subject             WHERE name = 'Super'             AND (resource = 0))  OR tasks.owner IN (SELECT *                     FROM super_on_users)  OR EXISTS (SELECT id FROM permissions_subject  WHERE resource = tasks.id  AND resource_type = 'task'  AND resource_location = 0  AND (t ()))) AND hidden = 0 AND usage_type = 'scan'   GROUP BY uuid  )      AS agg_sub GROUP BY outer_group_column, outer_subgroup_column ORDER BY outer_group_column DESC, max (text_column_0) DESC LIMIT 10 OFFSET 0;
gsa_1                  | gsad  gmp:MESSAGE:2022-11-25 18h13.35 GMT:13: Authentication success for 'admin' from 192.168.18.9
gsa_1                  | libgvm util:WARNING:2022-11-25 18h14.19 GMT:13:    End error: Error on line 1 char 1: Document was empty or contained only whitespace
gvmd_1                 | md   main:WARNING:2022-11-25 18h14.20 UTC:7270: write_to_client_unix: failed to write to client: Broken pipe
mqtt-broker_1          | 1669478209: Client 3576ba88-02c9-4585-b236-45922a0a19c3 has exceeded timeout, disconnecting.
gsa_1                  | gsad  gmp:MESSAGE:2022-11-27 22h43.10 GMT:13: Authentication success for 'admin' from 192.168.18.9
gvmd_1                 | event task:MESSAGE:2022-11-27 22h45.52 UTC:26438: Status of task vip-staging internal hosts (1b97362e-9a49-48cc-b29c-2dd8e81aa468) has changed to Requested
gvmd_1                 | event task:MESSAGE:2022-11-27 22h45.52 UTC:26438: Task vip-staging internal hosts (1b97362e-9a49-48cc-b29c-2dd8e81aa468) has been requested to start by admin
gsa_1                  | libgvm util:WARNING:2022-11-27 22h45.54 GMT:13:    End error: Error on line 1 char 1: Document was empty or contained only whitespace
pg-gvm_1               | 2022-11-27 22:45:55.379 UTC [30276] gvmd@gvmd ERROR:  canceling statement due to user request
pg-gvm_1               | 2022-11-27 22:45:55.379 UTC [30276] gvmd@gvmd CONTEXT:  SQL function "report_severity_count" statement 1
pg-gvm_1               | 2022-11-27 22:45:55.379 UTC [30276] gvmd@gvmd STATEMENT:  SELECT sum(aggregate_count) AS outer_count, aggregate_group_value AS outer_group_column, aggregate_subgroup_value AS outer_subgroup_column , max (text_column_0), max (text_column_1), max (text_column_2), max (text_column_3) FROM (WITH permissions_subject     AS (SELECT * FROM permissions         WHERE subject_location               = 0         AND ((subject_type = 'user'               AND subject                   = (SELECT id FROM users WHERE users.uuid = '933ce1ed-06f4-471e-aa20-debfd2bd76d6'))              OR (subject_type = 'group'                  AND subject                      IN (SELECT DISTINCT "group"              FROM group_users                          WHERE "user"                                = (SELECT id FROM users WHERE users.uuid = '933ce1ed-06f4-471e-aa20-debfd2bd76d6')))              OR (subject_type = 'role'   AND subject                      IN (SELECT DISTINCT role                          FROM role_users                          WHERE "user"                                = (SELECT id FROM users WHERE users.uuid = '933ce1ed-06f4-471e-aa20-debfd2bd76d6'))))),     super_on_users     AS (SELECT DISTINCT *         FROM (SELECT resource FROM permissions_subject               WHERE name = 'Super'               AND resource_type = 'user'               UNION               SELECT"user" FROM role_users               WHERE role                     IN (SELECT resource                         FROM permissions_subject                         WHERE name = 'Super'                         AND resource_type = 'role')          UNION               SELECT "user" FROM group_users               WHERE "group"                     IN (SELECT resource                         FROM permissions_subject                         WHERE name = 'Super'         AND resource_type = 'group'))              AS all_users)SELECT  count(*) AS aggregate_count, uuid AS aggregate_group_value, CAST (NULL AS TEXT) AS aggregate_subgroup_value, max (CASE WHEN target IS null OR opts.ignore_severity != 0 THEN 0 ELSE coalesce (report_severity_count (task_last_report (id),                                 opts.override, opts.min_qod,                                 'High') * 1.0            / nullif (report_result_host_count (task_last_report (id),                                                opts.min_qod), 0),          0) END) as text_column_0, max (modification_time) as text_column_1, max (name) as text_column_2, max (task_severity (id, opts.override, opts.min_qod)) as text_column_3 FROM tasks, (SELECT   0 AS override,   70 AS min_qod,   0 AS ignore_severity)  AS opts WHERE  ((tasks.owner   = (SELECT id FROM users WHERE users.uuid = '933ce1ed-06f4-471e-aa20-debfd2bd76d6'))  OR EXISTS (SELECT * FROM permissions_subject             WHERE name = 'Super'             AND (resource = 0))  OR tasks.owner IN (SELECT *                     FROM super_on_users)  OR EXISTS (SELECT id FROM permissions_subject  WHERE resource = tasks.id  AND resource_type = 'task'  AND resource_location = 0  AND (t ()))) AND hidden = 0 AND usage_type = 'scan'   GROUP BY uuid  )      AS agg_sub GROUP BY outer_group_column, outer_subgroup_column ORDER BY outer_group_column DESC, max (text_column_0) DESC LIMIT 10 OFFSET 0;