At 23:15 last night the following lines were logged in storm-backend.log in storm-cms.cr.cnaf.infn.it:

23:15:51.875 - ERROR [pool-4-thread-35] - SQL Error: The last packet successfully received from the server was 51,556,061 milliseconds ago. The last packet sent successfully to the server was 51,556,061 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem., SQLState: 08S01, VendorError: 0.
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 51,556,061 milliseconds ago. The last packet sent successfully to the server was 51,556,061 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.

After this message and many similar ones, the backend stood active but it basically stopped working.

We got several alarms from Sensu checks trying to perform a PtP but failing, and in the morning we realized gemss could not talk to the rest endpoint.

A restart solved the problem, as already happened in the past (SL6, older backend back then) and precisely for storm-cms.

We now have additional backend metrics which get logged and therefore can be monitored (see attachments).