PXB-3674 [DOCS] [feedback] Throttling backups - Provide clearer language. Provide a more user-friendly info flow. Provide a range of better code/functionality examples

[alina-derkach-oaza]

modified: docs/throttling-backups.md

[markusalbe]

Some quick notes:

1. We normally favour pidstat -d over iotop, as it is more commonly found already installed in most systems

2. Either pidstat or iotop require root privileges to function "correctly"; pidstat will only show the user's own processes, and iotop will emit error:

The Linux kernel interfaces that iotop relies on now require root priviliges or the NET_ADMIN capability.

3. Metric %util from iostat is important but does NOT indicate saturation. Rather an average queue size (aqu-sz) above 1 indicates saturation. Imagine an elevator that fits 10 people. It could be busy 100% of the time (%util), but only have 2 persons inside (aqu-sz), so it is not saturated, it is just busy. The await is how long it takes the elevator to go to a floor, in average.

4. There's incomplete example:
   xtrabackup --stream=xbstream | pv | gzip > backup.xb.gz
   At least it should include --backup and --throttle=N; People will often tend to mindlessly copy-paste, so is best to have fully functional examples that can be copy-paste and "always" work;
   Also, if all you want to do is test the rate of xtrabackup, then no need to compress and write to disk; Just do:
   xtrabackup --backup --throttle=2 --stream=xbstream 2>/dev/null| pv | cat - > /dev/null;
   That will purely test the throttling, as there's no possibility of the process being throttled by the single-threaded gzip.
   The 2>/dev/null before the first pipe is to avoid regular xtrabackup output, which would make the pv output unreadable.

5. Using --throttle will expose user to a known problem, where writes to the redo log happen faster than what XtraBackup can read them, and the redo log is a circular buffer with limited size, so it wraps around, causing XtraBackup to fail when the wrap around happens, with error similar to:

xtrabackup: error: expected log block no. 660048145, but got no. 661096713 from the log file.

xtrabackup: error: it looks like InnoDB log has wrapped around before xtrabackup could process all records due to either log copying being too slow, or log files being too small.

181121 17:32:05 [01] …done

xtrabackup: Error: xtrabackup_copy_logfile() failed```

The exact strings might change across versions, but I'm pretty sure it's always `xtrabackup_copy_logfile()`
To prevent this there are two options:

**1) Increase redo log size:**
```SET GLOBAL innodb_redo_log_capacity=<make it 2x current size>;```   
This can cause longer crash recovery times, but will provide better write performance for the application and will allow for throttled backups to complete.

**2) Use `--register-redo-log-consumer`** to prevent the wrap around, but this will block the writes to the database until XtraBackup can catch up with the redo log writer.  This can be useful if the backup is being produced on a replica, and its acceptable for the replication thread to fall behind.