ログ
なんかサービスが起動出来ないなと思ってよくよく見てみたらファイルシステムが読み込み専用でマウントされているみたいだった。このせいでログファイルが開けなくて怒られる。
$ touch hoge touch: cannot touch `hoge': 読み込み専用ファイルシステムです
/dev/sda3でマウントしてるっぽい。
# df -h Filesystem サイズ 使用 残り 使用% マウント位置 /dev/sda3 284G 262G 7.3G 98% / /dev/sda1 99M 12M 82M 13% /boot none 4.0G 0 4.0G 0% /dev/shm
というわけでdmesg内で"sda3"を確認してみる。
# dmesg | grep -n sda3 214: sda: sda1 sda2 sda3 221:EXT3-fs: sda3: orphan cleanup on readonly fs 228:EXT3-fs: sda3: 6 orphan inodes deleted 386:EXT3 FS on sda3, internal journal 400:EXT3-fs error (device sda3): ext3_free_blocks_sb: bit already cleared for block 33126379 401:Aborting journal on device sda3. 403:EXT3-fs error (device sda3): ext3_journal_start_sb: Detected aborted journal 405:EXT3-fs error (device sda3) in start_transaction: Journal has aborted 406:EXT3-fs error (device sda3) in ext3_reserve_inode_write: Journal has aborted 407:EXT3-fs error (device sda3) in ext3_truncate: Journal has aborted 408:EXT3-fs error (device sda3) in ext3_reserve_inode_write: Journal has aborted 409:EXT3-fs error (device sda3) in ext3_orphan_del: Journal has aborted 410:EXT3-fs error (device sda3) in ext3_reserve_inode_write: Journal has aborted 411:EXT3-fs error (device sda3) in ext3_delete_inode: Journal has aborted 412:EXT3-fs error (device sda3) in start_transaction: Journal has aborted 417:EXT3-fs error (device sda3) in start_transaction: Journal has aborted 418:EXT3-fs error (device sda3) in start_transaction: Journal has aborted 419:EXT3-fs error (device sda3) in start_transaction: Journal has aborted
うっわー、ext3のエラー出まくってる。とりあえず210行目らへんを見てみる。
... sda: cache data unavailable sda: assuming drive cache: write through SCSI device sda: 629145600 512-byte hdwr sectors (322123 MB) sda: cache data unavailable sda: assuming drive cache: write through sda: sda1 sda2 sda3 Attached scsi disk sda at scsi0, channel 0, id 0, lun 0 Fusion MPT SAS Host driver 3.12.19.00rh libata version 2.00 loaded. EXT3-fs: INFO: recovery required on readonly filesystem. EXT3-fs: write access will be enabled during recovery. kjournald starting. Commit interval 5 seconds EXT3-fs: sda3: orphan cleanup on readonly fs ext3_orphan_cleanup: deleting unreferenced inode 35946504 ext3_orphan_cleanup: deleting unreferenced inode 35946503 ext3_orphan_cleanup: deleting unreferenced inode 36159521 ext3_orphan_cleanup: deleting unreferenced inode 36159520 ext3_orphan_cleanup: deleting unreferenced inode 36683784 ext3_orphan_cleanup: deleting unreferenced inode 35946500 EXT3-fs: sda3: 6 orphan inodes deleted EXT3-fs: recovery complete. EXT3-fs: mounted filesystem with ordered data mode. SELinux: Disabled at runtime. SELinux: Unregistering netfilter hooks inserting floppy driver for 2.6.9-78.ELsmp Floppy drive(s): fd0 is 1.44M FDC 0 is a post-1991 82077 pcnet32.c:v1.31 29.04.2005 tsbogend@alpha.franken.de ACPI: PCI Interrupt 0000:02:01.0[A] -> GSI 19 (level, low) -> IRQ 193 pcnet32: PCnet/PCI II 79C970A at 0x2000, 00 0c 29 fe 59 7e assigned IRQ 193. divert: allocating divert_blk for eth0 eth0: registered as PCnet/PCI II 79C970A pcnet32: 1 cards_found. pciehp: acpi_pciehprm: Slot sun(20) at s:b:d:f=0x00:02:00:00 pciehp: acpi_pciehprm: Slot sun(21) at s:b:d:f=0x00:02:01:00 pciehp: acpi_pciehprm: Slot sun(22) at s:b:d:f=0x00:02:02:00 ...
inodeがぶっ壊れてて孤立してたから削除した後にrecovery終わったって言って普通にマウントしてるっぽい。これは再起動フラグ？というわけでrebootしてみた。もう一回dmesg見てみてダメだったらfsckする。まあ立ち上がらなければsingle user modeで。
と思って再起動したら案の定こけた。強制的にsingle user mode突入。
# fsck /dev/sda3
とりあえずこれでOSは起動できたけど、こっからDBのデータファイル復旧とか...萎えるよねえ。
