验证19c DG_BROKER快速启动故障切换FSFO

配置了dg_broker以后，默认的FSFO并未启动：
Fast-Start Failover: Disabled
这样如果主库意外故障，备库无法自动切换为主库，需要手动切换。
如果启动了FSFO，备库会自动切换为主库。以下是验证过程。

如果启动FSFO，需要主备库开启快速闪回功能，确认flashback_on为yes。

启动FSFO
DGMGRL> enable fast_start failover;
Enabled in Potential Data Loss Mode.

此时show configuration显示告警，没有打开observer，如下方式在备库执行打开observer
nohup dgmgrl sys/passwd@wydb "start observer file=FSFO.dat" >> fsfo.log 2>&1 &

观察监控日志：
tail -100f fsfo.log

模拟主库故障：
srvctl stop database -db wydb -stopoption ABORT

监控日志如下：

Connected to "wydb"
Connected as SYSDBA.
[W000 2020-11-28T18:32:51.689+08:00] FSFO target standby is wydbadg
Observer 'wydbadg' started
[W000 2020-11-28T18:32:51.850+08:00] Observer trace level is set to USER
[W000 2020-11-28T18:32:51.850+08:00] Try to connect to the primary.
[W000 2020-11-28T18:32:51.850+08:00] Try to connect to the primary wydb.
[W000 2020-11-28T18:32:51.869+08:00] The standby wydbadg is ready to be a FSFO target
[W000 2020-11-28T18:32:52.869+08:00] Connection to the primary restored!
[W000 2020-11-28T18:32:54.870+08:00] Disconnecting from database wydb.
[W000 2020-11-28T18:37:55.316+08:00] Primary database cannot be reached.
[W000 2020-11-28T18:37:55.316+08:00] Fast-Start Failover threshold has not exceeded. Retry for the next 30 seconds
[W000 2020-11-28T18:37:56.316+08:00] Try to connect to the primary.
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Unable to connect to database using wydb
[W000 2020-11-28T18:37:56.336+08:00] Primary database cannot be reached.
[W000 2020-11-28T18:37:57.336+08:00] Try to connect to the primary.
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Unable to connect to database using wydb
[W000 2020-11-28T18:37:57.339+08:00] Primary database cannot be reached.
[W000 2020-11-28T18:37:58.339+08:00] Try to connect to the primary.
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Unable to connect to database using wydb
[W000 2020-11-28T18:37:58.342+08:00] Primary database cannot be reached.
[W000 2020-11-28T18:37:59.342+08:00] Try to connect to the primary.
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Unable to connect to database using wydb
[W000 2020-11-28T18:38:22.425+08:00] Primary database cannot be reached.
[W000 2020-11-28T18:38:22.425+08:00] Fast-Start Failover threshold has not exceeded. Retry for the next 3 seconds
[W000 2020-11-28T18:38:23.425+08:00] Try to connect to the primary.
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Unable to connect to database using wydb
[W000 2020-11-28T18:38:23.428+08:00] Primary database cannot be reached.
[W000 2020-11-28T18:38:23.428+08:00] Fast-Start Failover threshold has not exceeded. Retry for the next 2 seconds
[W000 2020-11-28T18:38:24.428+08:00] Try to connect to the primary.
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Unable to connect to database using wydb
[W000 2020-11-28T18:38:24.432+08:00] Primary database cannot be reached.
[W000 2020-11-28T18:38:24.432+08:00] Fast-Start Failover threshold has not exceeded. Retry for the next 1 second
[W000 2020-11-28T18:38:25.432+08:00] Try to connect to the primary.
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Unable to connect to database using wydb
[W000 2020-11-28T18:38:25.435+08:00] Primary database cannot be reached.
[W000 2020-11-28T18:38:25.435+08:00] Fast-Start Failover threshold has expired.
[W000 2020-11-28T18:38:25.435+08:00] Try to connect to the standby.
[W000 2020-11-28T18:38:25.435+08:00] Making a last connection attempt to primary database before proceeding with Fast-Start Failover.
[W000 2020-11-28T18:38:25.435+08:00] Check if the standby is ready for failover.
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Unable to connect to database using wydb
[S002 2020-11-28T18:38:26.442+08:00] Fast-Start Failover started...

2020-11-28T18:38:26.442+08:00
Initiating Fast-Start Failover to database "wydbadg"...
[S002 2020-11-28T18:38:26.442+08:00] Initiating Fast-start Failover.
Performing failover NOW, please wait...
Failover succeeded, new primary is "wydbadg"
2020-11-28T18:39:01.453+08:00
[S002 2020-11-28T18:39:01.453+08:00] Fast-Start Failover finished...
[W000 2020-11-28T18:39:01.453+08:00] Failover succeeded. Restart pinging.
[W000 2020-11-28T18:39:01.461+08:00] Primary database has changed to wydbadg.

可见备库自动切换为主库，且原主库失联。

再次启动原主库，观察日志情况如下：
srvctl start database -d wydb

[W000 2020-11-28T18:39:01.464+08:00] Try to connect to the primary.
[W000 2020-11-28T18:39:01.464+08:00] Try to connect to the primary wydbadg.
[W000 2020-11-28T18:39:01.528+08:00] The standby wydb needs to be reinstated
[W000 2020-11-28T18:39:01.528+08:00] Try to connect to the new standby wydb.
[W000 2020-11-28T18:39:01.528+08:00] Connection to the primary restored!
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Unable to connect to database using wydb
[W000 2020-11-28T18:39:03.529+08:00] Disconnecting from database wydbadg.
[W000 2020-11-28T18:39:31.570+08:00] Try to connect to the new standby wydb.
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Unable to connect to database using wydb
[W000 2020-11-28T18:40:02.712+08:00] Try to connect to the new standby wydb.
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Unable to connect to database using wydb
[W000 2020-11-28T18:40:32.761+08:00] Try to connect to the new standby wydb.
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Unable to connect to database using wydb
[W000 2020-11-28T18:41:02.804+08:00] Try to connect to the new standby wydb.
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Unable to connect to database using wydb
[W000 2020-11-28T18:41:32.846+08:00] Try to connect to the new standby wydb.
[W000 2020-11-28T18:41:33.850+08:00] Connection to the new standby restored!
[W000 2020-11-28T18:42:07.900+08:00] Try to connect to the primary wydbadg.
[W000 2020-11-28T18:42:08.900+08:00] Connection to the primary restored!
[W000 2020-11-28T18:42:08.900+08:00] Wait for new primary to be ready to reinstate.
[W000 2020-11-28T18:42:08.904+08:00] New primary is now ready to reinstate.
[W000 2020-11-28T18:42:09.904+08:00] Issuing REINSTATE command.

2020-11-28T18:42:09.904+08:00
Initiating reinstatement for database "wydb"...
Reinstating database "wydb", please wait...
[W000 2020-11-28T18:42:50.963+08:00] The standby wydb is ready to be a FSFO target
Reinstatement of database "wydb" succeeded
2020-11-28T18:43:50.572+08:00
[W000 2020-11-28T18:43:51.046+08:00] Successfully reinstated database wydb.

可见DG_BROKER主动连接到原主库，并将原主库初始化为备库。

以下再切回原来主备库状态，关闭现在的主库：
shutdown abort
观察日志如下：

[W000 2020-11-28T19:51:05.162+08:00] Primary database cannot be reached.
[W000 2020-11-28T19:51:05.163+08:00] Fast-Start Failover threshold has not exceeded. Retry for the next 30 seconds
[W000 2020-11-28T19:51:06.163+08:00] Try to connect to the primary.
[W000 2020-11-28T19:51:07.219+08:00] Primary database cannot be reached.
[W000 2020-11-28T19:51:08.219+08:00] Try to connect to the primary.
[W000 2020-11-28T19:51:33.932+08:00] Primary database cannot be reached.
[W000 2020-11-28T19:51:33.932+08:00] Fast-Start Failover threshold has not exceeded. Retry for the next 2 seconds
[W000 2020-11-28T19:51:34.933+08:00] Try to connect to the primary.
[W000 2020-11-28T19:51:35.986+08:00] Primary database cannot be reached.
[W000 2020-11-28T19:51:35.987+08:00] Fast-Start Failover threshold has expired.
[W000 2020-11-28T19:51:35.987+08:00] Try to connect to the standby.
[W000 2020-11-28T19:51:35.987+08:00] Making a last connection attempt to primary database before proceeding with Fast-Start Failover.
[W000 2020-11-28T19:51:35.987+08:00] Check if the standby is ready for failover.
[S005 2020-11-28T19:51:37.000+08:00] Fast-Start Failover started...

2020-11-28T19:51:37.000+08:00
Initiating Fast-Start Failover to database "wydb"...
[S005 2020-11-28T19:51:37.000+08:00] Initiating Fast-start Failover.
Performing failover NOW, please wait...
Failover succeeded, new primary is "wydb"
2020-11-28T19:52:11.606+08:00
[S005 2020-11-28T19:52:11.606+08:00] Fast-Start Failover finished...
[W000 2020-11-28T19:52:11.606+08:00] Failover succeeded. Restart pinging.
[W000 2020-11-28T19:52:11.617+08:00] Primary database has changed to wydb.
[W000 2020-11-28T19:52:11.620+08:00] Try to connect to the primary.
[W000 2020-11-28T19:52:11.620+08:00] Try to connect to the primary wydb.
[W000 2020-11-28T19:52:11.777+08:00] Connection to the primary restored!
[W000 2020-11-28T19:52:11.786+08:00] The standby wydbadg needs to be reinstated
[W000 2020-11-28T19:52:11.786+08:00] Try to connect to the new standby wydbadg.
[W000 2020-11-28T19:52:13.787+08:00] Connection to the new standby restored!
[W000 2020-11-28T19:52:13.787+08:00] Disconnecting from database wydb.
[W000 2020-11-28T19:52:14.794+08:00] Failed to ping the new standby.

可见原主库顺利切换为新的主库。

再次打开刚刚关闭的库：

SYS@wydb1> startup
ORACLE instance started.

Total System Global Area    137438951616 bytes
Fixed Size                      30398656 bytes
Variable Size                16106127360 bytes
Database Buffers            121064390656 bytes
Redo Buffers                   238034944 bytes
Database mounted.
ORA-16649: possible failover to another database prevents this database from being opened

可见库不允许打开，让DG_BROKER后端初始化状态，日志如下：

[W000 2020-11-28T19:56:02.289+08:00] Wait for new primary to be ready to reinstate.
[W000 2020-11-28T19:56:03.295+08:00] New primary is now ready to reinstate.
[W000 2020-11-28T19:56:04.295+08:00] Issuing REINSTATE command.

2020-11-28T19:56:04.295+08:00
Initiating reinstatement for database "wydbadg"...
Reinstating database "wydbadg", please wait...
[W000 2020-11-28T19:56:26.342+08:00] The standby wydbadg is ready to be a FSFO target
Reinstatement of database "wydbadg" succeeded
2020-11-28T19:58:00.661+08:00
[W000 2020-11-28T19:58:01.561+08:00] Successfully reinstated database wydbadg.

可见新打开的库状态被初始化，状态如下：

SYS@wydb1> select name,open_mode,database_role,db_unique_name,protection_mode from v$database;

NAME                 OPEN_MODE            DATABASE_ROLE    DB_UNIQUE_NAME                 PROTECTION_MODE
-------------------- -------------------- ---------------- ------------------------------ --------------------
WYDB                 READ ONLY WITH APPLY PHYSICAL STANDBY wydbadg                        MAXIMUM PERFORMANCE

即，新打开的库再次成为原来的备库状态。