openstack集群中發(fā)現(xiàn)某計(jì)算節(jié)點(diǎn)計(jì)算服務(wù)down
在計(jì)算節(jié)點(diǎn)輸入:
service nova-compute restart
服務(wù)無(wú)法啟動(dòng)餐弱,查看:
/var/log/nova/nova-compute.log
發(fā)現(xiàn)如下錯(cuò)誤:
InstanceNotFound: Instance 00f71bfc-b8c5-474c-94c8-02686c1af24b could not be found
說(shuō)明出現(xiàn)了僵尸實(shí)例宴霸,可能是使用虛擬機(jī)時(shí)非法關(guān)閉nova服務(wù)或關(guān)閉虛擬機(jī)導(dǎo)致。解決辦法是刪除該實(shí)例膏蚓。
首先出現(xiàn)該問(wèn)題的計(jì)算節(jié)點(diǎn)上找到如下目錄:
/var/lib/nova/instacnes
刪除該實(shí)例文件:
sudo rm -rf 00f71bfc-b8c5-474c-94c8-02686c1af24b
然后在控制節(jié)點(diǎn)數(shù)據(jù)庫(kù)中刪除該實(shí)例,以下語(yǔ)句中
$1 代表 00f71bfc-b8c5-474c-94c8-02686c1af24b:
sudo mysql -u root -p
use nova;
delete from security_group_instance_association where instance_uuid='$1';
delete from instance_info_caches where instance_uuid='$1';
delete from block_device_mapping where instance_uuid='$1';
delete from instance_actions where instance_uuid='$1'; //注意瓢谢,這里需要手動(dòng)刪除另外的記錄,下文會(huì)提到
delete from instance_faults where instance_uuid='$1';
delete from instance_extra where instance_uuid='$1';
delete from instance_system_metadata where instance_uuid='$1';
delete from instances where uuid='$1'; //注意驮瞧,在我使用的pike版本中氓扛,instances 表中為 uuid,而不是 instance_id
在如下語(yǔ)句中:
delete from instance_actions where instance_uuid='$1';
提示:
Cannot delete or update a parent row: a foreign key constraint fails('nova'.'instance_actions_events', CONSTRAINT 'instance_actions_events_ibfk_1' FOREIGN KEY ('action_id') REFERENCES 'instance_actions'('id'))
因?yàn)?instance_actions 表中 主鍵id 是 instance_actions_events 表中 action_id 的外鍵论笔,如果刪除 instance_actions 表中記錄采郎,且該記錄中的id主鍵被 instance_actions_events 表中 action_id 引用,就會(huì)報(bào)錯(cuò)狂魔。
所以先判斷刪除 instance_action_events 表中的數(shù)據(jù)蒜埋。
首先根據(jù) uuid 找到 instance_actions 表中的記錄:
select id from instance_actions where instance_uuid='$1';
記錄找到的id,然后去 instance_actions_events 表中刪除記錄,假設(shè)id=$2:
delete from instance_actions_events where action_id='$2';
最后執(zhí)行:
delete from instance_actions where instance_uuid='$1';
成功刪除最楷。
在以下語(yǔ)句時(shí):
delete from instances where uuid='$1';
遇到類(lèi)似問(wèn)題:
Cannot delete or update a parent row: a foreign key constraint fails ('nova'.'migrations', CONSTRAINT 'fk_migrations_instance_uuid' FOREIGH KEY ('instance_uuid') REFERENCES 'instances('uuid')')
看來(lái)這次產(chǎn)生僵尸實(shí)例是虛擬機(jī)遷移出錯(cuò)整份,所以在 migrations 表中刪除記錄:
delete from migrations where instance_uuid='$1';
再次執(zhí)行:
delete from instances where uuid='$1';
遇到相似提示:
Cannot delete or update a parent row: a foreign key constraint fails ('nova'.'virtual_interfaces', CONSTRAINT 'virtual_interfaces_instance_uuid' FOREIGH KEY ('instance_uuid') REFERENCES 'instances('uuid')')
執(zhí)行:
delete from virtual_interfaces where instance_uuid='$1';
最后執(zhí)行:
delete from instances where uuid='$1';
成功刪除記錄!
在出錯(cuò)的計(jì)算節(jié)點(diǎn)中運(yùn)行:
service nova-compute restart
service nova-compute status
發(fā)現(xiàn)服務(wù)成功啟動(dòng)管嬉,錯(cuò)誤解決皂林。
更新
由于手動(dòng)操作太麻煩,于是針對(duì)實(shí)驗(yàn)情況寫(xiě)了自動(dòng)刪除腳本蚯撩,如下:
sudo mysql -u root -e "select id from instance_actions where instance_uuid='$1'" nova > tmp.txt
for line in $(cat tmp.txt)
do
if [ $line -gt 0 ] 2>/dev/null
then
echo $line
sudo mysql -u root -e "delete from instance_actions_events where action_id='$line'" nova
fi
done
sudo mysql -u root -e "delete from instance_actions where instance_uuid='$1'" nova
sudo mysql -u root -e "delete from security_group_instance_association where instance_uuid='$1'" nova
sudo mysql -u root -e "delete from instance_info_caches where instance_uuid='$1'" nova
sudo mysql -u root -e "delete from block_device_mapping where instance_uuid='$1'" nova
sudo mysql -u root -e "delete from instance_faults where instance_uuid='$1'" nova
sudo mysql -u root -e "delete from instance_extra where instance_uuid='$1'" nova
sudo mysql -u root -e "delete from instance_system_metadata where instance_uuid='$1'" nova
sudo mysql -u root -e "delete from migrations where instance_uuid='$1'" nova
sudo mysql -u root -e "delete from virtual_interfaces where instance_uuid='$1'" nova
sudo mysql -u root -e "delete from instances where uuid='$1'" nova
該腳本可自動(dòng)刪除由于遷移產(chǎn)生僵尸實(shí)例的問(wèn)題(不適用于所有情況)