If a CephFS journal has become damaged, expert intervention may be required to restore the filesystem to a working state.
The cephfs-journal-tool utility provides functionality to aid experts in examining, modifying, and extracting data from journals.
Warning
This tool is dangerous because it directly modifies internal data structures of the filesystem. Make backups, be careful, and seek expert advice. If you are unsure, do not run this tool.
cephfs-journal-tool journal <inspect|import|export|reset>
cephfs-journal-tool header <get|set>
cephfs-journal-tool event <get|splice|apply> [filter] <list|json|summary>
The tool operates in three modes: journal, header and event, meaning the whole journal, the header, and the events within the journal respectively.
This should be your starting point to assess the state of a journal.
# cephfs-journal-tool journal inspect
Overall journal integrity: DAMAGED
Objects missing:
0x1
Corrupt regions:
0x400000-ffffffffffffffff
# cephfs-journal-tool journal export myjournal.bin
journal is 4194304~80643
read 80643 bytes at offset 4194304
wrote 80643 bytes at offset 4194304 to myjournal.bin
NOTE: this is a _sparse_ file; you can
$ tar cSzf myjournal.bin.tgz myjournal.bin
to efficiently compress it while preserving sparseness.
# cephfs-journal-tool journal import myjournal.bin
undump myjournal.bin
start 4194304 len 80643
writing header 200.00000000
writing 4194304~80643
done.
Note
It is wise to use the journal export <backup file> command to make a journal backup before any further manipulation.
# cephfs-journal-tool header get
{ "magic": "ceph fs volume v011",
"write_pos": 4274947,
"expire_pos": 4194304,
"trimmed_pos": 4194303,
"layout": { "stripe_unit": 4194304,
"stripe_count": 4194304,
"object_size": 4194304,
"cas_hash": 4194304,
"object_stripe_unit": 4194304,
"pg_pool": 4194304}}
# cephfs-journal-tool header set trimmed_pos 4194303
Updating trimmed_pos 0x400000 -> 0x3fffff
Successfully updated header.
Event mode allows detailed examination and manipulation of the contents of the journal. Event mode can operate on all events in the journal, or filters may be applied.
The arguments following cephfs-journal-tool event consist of an action, optional filter parameters, and an output mode:
cephfs-journal-tool event <action> [filter] <output>
Actions:
Filtering:
Filters may be combined on an AND basis (i.e. only the intersection of events from each filter).
Output modes:
# cephfs-journal-tool event get json --path output.json
Wrote output to JSON file 'output.json'
# cephfs-journal-tool event get summary
Events by type:
NOOP: 2
OPEN: 2
SESSION: 2
SUBTREEMAP: 1
UPDATE: 43
# cephfs-journal-tool event get list
0x400000 SUBTREEMAP: ()
0x400308 SESSION: ()
0x4003de UPDATE: (setattr)
/
0x40068b UPDATE: (mkdir)
diralpha
0x400d1b UPDATE: (mkdir)
diralpha/filealpha1
0x401666 UPDATE: (unlink_local)
stray0/10000000001
diralpha/filealpha1
0x40228d UPDATE: (unlink_local)
diralpha
stray0/10000000000
0x402bf9 UPDATE: (scatter_writebehind)
stray0
0x403150 UPDATE: (mkdir)
dirbravo
0x4037e0 UPDATE: (openc)
dirbravo/.filebravo1.swp
0x404032 UPDATE: (openc)
dirbravo/.filebravo1.swpx
# cephfs-journal-tool event get --path /filebravo1 list
0x40785a UPDATE: (openc)
dirbravo/filebravo1
0x4103ee UPDATE: (cap update)
dirbravo/filebravo1
# cephfs-journal-tool event splice --range 0x40f754..0x410bf1 summary
Events by type:
OPEN: 1
UPDATE: 2
# cephfs-journal-tool event apply --range 0x410bf1.. summary
Events by type:
NOOP: 1
SESSION: 1
UPDATE: 9
# cephfs-journal-tool event get --inode=1099511627776 list
0x40068b UPDATE: (mkdir)
diralpha
0x400d1b UPDATE: (mkdir)
diralpha/filealpha1
0x401666 UPDATE: (unlink_local)
stray0/10000000001
diralpha/filealpha1
0x40228d UPDATE: (unlink_local)
diralpha
stray0/10000000000
# cephfs-journal-tool event get --frag=1099511627776 --dname=filealpha1 list
0x400d1b UPDATE: (mkdir)
diralpha/filealpha1
0x401666 UPDATE: (unlink_local)
stray0/10000000001
diralpha/filealpha1
# cephfs-journal-tool event get binary --path bin_events
Wrote output to binary files in directory 'bin_events'