Troubleshooting Bravura Security Fabric server failures

The Backlog size value is a measure of the replication data that is pending and waiting to be synchronized across the nodes.

As long as there is data in the replication backlogs, the nodes show different data in their web UI as a temporary synchronization issue.

If stored procedures (sprocs) fail to be applied on some nodes while they succeed on others, the sync issue becomes more permanent.

Some sprocs that fail once may succeed later if the issue that caused the original failure is resolved. For example, when a WstnPwdCheckOut fails with "stored procedure exception" = "No lock to check out password", and a user performs a successful check-in on the affected account on the node where that sproc failed, the next WstnPwdCheckOut on the same account will succeed on that node as well.

Queue delay represents the time difference between when a replication event is generated on the source node and when it is actually processed on the destination node. This delay can indicate communication issues between nodes, such as network latency or problems with the database server that may be slowing down the replication process. For example, if it grows, the iddb on the observed node is having trouble communicating with the node listed on the page with the increased delay.

The Queue size refers to the amount of data waiting to be replicated from one node to another within the system. It is a measure of the replication data that has been generated but not yet processed or sent to the destination node. A consistently large or growing queue size may indicate issues with network connectivity, database performance, sproc issues, or other system bottlenecks that could affect the efficiency of data replication.

Send queues grow when the iddb service on the observed node has trouble connecting to the iddb of the node in the title of the specific page section.

Possible connection issues include network latency, a slow-running iddb on the sending or receiving end, or handle or socket exhaustion on the sending or receiving node.

In rare cases:

Receive queues grow when there is a problem on the backend database of the receiving node.

All receive queues tend to grow at once, with similar values, if all the nodes are under load.

Possible database issues include:

If this value increases, a sproc is likely taking a long time to run on that database.

As of Bravura Security Fabric version 12.5.0 the iddb service saves a "snapshot" of what can be seen in the Database replication details pages per node, in telemetry (*.rst) files located in the instance's db\ directory, with one record every 30 seconds.

Those files can be parsed in many ways for the rich data present in them. For example to get a trace of replication delays in seconds, you could use the command:

grep "2024-0[67]" telemetry-recv-bravura1.local_hipam.rst | awk -F'|' '{print $1,",",$5}'|sed s/ie=//|sed s/qd=//>times-FTC706.csv

In the above command:

You can then use a spreadsheet program to render a chart from the two data columns, like this:

These are the counts of entries with a worst-sproc duration of six or more digits in milliseconds (matching [0-9]{6,}), for those two months. These counts are not complete, unlike those collected from stored-procedure performance entries in idmsuite.log, because thousands of sprocs can run in 30 seconds and only one or two are recorded in these telemetry files:

egrep -c "2024-0[67].*wpt=[0-9]{6,}" telemetry-recv-PTC-bravura1.local_hipam.rst
4534

... and itemized by counts of sprocs:

egrep "2024-0[67].*wpt=[0-9]{6,}" telemetry-recv-bravura1.local_hipam.rst|egrep -o
      "wp=[^|]+"|sort|uniq -c|sort -k1,1nr
   
3491 wp=PolicyRuleComputerPrepare
388 wp=ManageRuleComputerPrepare
335 wp=WstnPwdReset
264 wp=WstnPwdUpdateStatus
34 wp=DashWorkStatsCompute
13 wp=WstnPwdQueueRandomize
5 wp=WstnPwdResetFail
3 wp=LoaddbInit

These are the fields present in each entry of the telemetry files:

If the above is happening:

If the values in the web UI are decreasing, it means that the queues grew while database administrators (or automation) were performing maintenance on the servers, and now they are flushing.

If possible, Bravura Security Fabric should be installed onto its own, dedicated drive.

To maximize available free disk space, limit or remove unnecessary files on the Bravura Security Fabric instance’s installation drive.

To prevent the instance from running out of disk space in the first place, ensure that e-mail notifications are working, and that a suitable e-mail address or addresses is configured RECIPIENT EMAIL variable, under Manage the system > Workflow > E-mail configuration.

During an outage limited to one node, non-outage nodes will queue database updates to disk until contact is re-established with the outage nodes. If the outage is prolonged, those queues will eventually start to grow. With enough time, disk space will remain depleted until the free-space threshold is reached. At that point, the node will enter DB COMMIT SUSPEND mode (to safeguard against data loss), making the node unusable until the issue is addressed. This is normal, documented behavior.

When the iddb service is stopped unexpectedly (it crashes or the server it runs on loses power) some of the old queues may remain in place and take space without them being used.

Replication queue growth is one cause of disk space exhaustion. Several other factors can also consume free disk space on a server where Bravura Security Fabric is installed, particularly when the backend database is on the same server as the application: