Lines Matching full:metadata
68 Metadata directly supporting these functions (e.g. files, directories, space
69 mappings) are sometimes called primary metadata.
70 Secondary metadata (e.g. reverse mapping and directory parent pointers) support
73 Summary metadata, as the name implies, condense information contained in
74 primary metadata for performance reasons.
76 The filesystem check (fsck) tool examines all the metadata in a filesystem
78 In addition to looking for obvious metadata corruptions, fsck also
79 cross-references different types of metadata records with each other to look
84 the filesystem metadata to a consistent state, not to maximize the data
91 More recent filesystem designs contain enough redundancy in their metadata that
124 It walks all metadata in the filesystem looking for inconsistencies in the
125 metadata, though it lacks any ability to repair what it finds.
134 while it scans the metadata of the entire filesystem.
136 inconsistencies in file metadata and directory tree by erasing things as needed
138 Space usage metadata are rebuilt from the observed file metadata.
146 shutdowns occur as a result of silent corruptions in the metadata.
176 metadata, an in-kernel facility to repair metadata, and a userspace driver
190 | The kernel portion of online fsck that validates metadata is called |
191 | "online scrub", and portion of the kernel that fixes metadata is called |
204 Even if a piece of filesystem metadata can only be regenerated by scanning the
209 metadata to enable targeted checking and repair operations while the system
217 Because it is necessary for online fsck to lock and scan live metadata objects,
220 for identifying individual metadata items, scheduling work items for them,
235 metadata structure, and handle it well.
272 2. Check allocation group metadata, all realtime volume metadata, and all quota
274 Each metadata structure is scheduled as a separate scrub item.
283 3. Check all metadata of every file in the filesystem.
284 Each metadata structure is also scheduled as a separate scrub item.
301 5. By the start of this phase, all primary and secondary filesystem metadata
325 the one aspect of a metadata object represented by a scrub item:
338 Repair functions generally choose rebuild a structure from other metadata
344 3. In the third step, the kernel runs the same checks over the new metadata
348 Classification of Metadata
351 Each type of metadata object (and therefore each type of scrub item) is
354 Primary Metadata argument
357 Metadata structures in this category should be most familiar to filesystem
379 Primary metadata objects are the simplest for scrub to process.
383 errors and cross-references healthy records against other metadata to look for
387 The repair function scans available metadata as needed to record all the
412 Most primary metadata repair functions stage their intermediate results in an
421 Secondary Metadata argument
424 Metadata structures in this category reflect records found in primary metadata,
427 Secondary metadata include:
433 This class of metadata is difficult for scrub to process because scrub attaches
434 to the secondary object but needs to check primary metadata, which runs counter
437 metadata.
439 Repairs, however, require a full scan of primary metadata, which can take a
471 primary metadata, but doing so would make it massively more complex and less
477 Inspiration for the secondary metadata repair strategy was drawn from section
509 employed these live scans to build a shadow copy of the metadata and then
519 Metadata structures in this last category summarize the contents of primary
520 metadata records.
522 smaller than the primary metadata which they represent.
569 - **Decreased performance**: Adding metadata indices to the filesystem
589 If the keyspaces of several metadata indices overlap in some manner but a
628 1. Detect inconsistencies in the metadata;
641 systematic testing of every attribute of every type of metadata object.
663 alignment about what constitutes consistent metadata.
669 ``xfs_repair`` to rebuild the filesystem's metadata indices between tests.
678 General Fuzz Testing of Metadata Blocks
684 to test the rather common fault that entire metadata blocks get corrupted.
686 containing every possible type of metadata object.
688 a single block of a specific type of metadata object, trash it with the
690 particular metadata validation strategy.
694 detect and eliminate the inconsistent metadata.
700 * For each metadata object existing on the filesystem:
706 1. The kernel verifiers to stop obviously bad metadata
710 Targeted Fuzz Testing of Metadata Records
715 of every metadata field of every metadata object in the filesystem.
716 ``xfs_db`` can modify every field of every metadata structure in every
720 containing every metadata format known to the filesystem, ``xfs_db`` can be
725 * For each metadata object existing on the filesystem...
727 * For each record inside that metadata object...
744 1. The kernel verifiers to stop obviously bad metadata
757 classes of metadata objects in ``xfs_repair``.
781 inconsistencies into the filesystem metadata, and regular workloads should
801 any unexpected filesystem shutdowns due to corrupted metadata, kernel hang
822 metadata in a filesystem, ``xfs_scrub`` can be run as a foreground process on
824 The program checks every piece of metadata in the filesystem while the
898 inconsistencies are detected in the filesystem metadata during regular
926 code that provide the ability to check and repair metadata while the system
929 foundation for checking metadata.
933 Self Describing Metadata
944 The first three components enable checking tools to disregard alleged metadata
960 the filesystem to detect obvious corruption when reading metadata blocks from
962 between metadata structures.
965 Documentation/filesystems/xfs-self-describing-metadata.rst
978 metadata, or more space-efficient encoding techniques.
985 file metadata (the directory tree, the file block map, and the allocation
1006 | As for metadata, the complexity of adding a new secondary index of space |
1027 The owner field tells scrub which metadata structure or file inode have been
1035 Online filesystem checking judges the consistency of each primary metadata
1053 the above primary metadata are in doubt.
1054 The checking code for most primary metadata follows a path similar to the
1057 2. Proving the consistency of secondary metadata with the primary metadata is
1058 difficult because that requires a full scan of all primary space metadata,
1076 metadata.
1083 The first step of checking a metadata structure is to examine every record
1087 metadata from wreaking havoc on the system.
1089 three decisions about the health of a metadata structure:
1097 metadata (``XFS_SCRUB_OFLAG_PREEN``) ?
1101 The following sections describe how the metadata scrubbing process works.
1103 Metadata Buffer Verification
1106 The lowest layer of metadata protection in XFS are the metadata verifiers built
1114 This assumes that metadata blocks only have one owner, which is always true
1129 corrupt metadata force the cancellation of a dirty transaction.
1131 Every online fsck scrubbing function is expected to read every ondisk metadata
1142 After the buffer cache, the next level of metadata protection is the internal
1177 Various pieces of filesystem metadata are directly controlled by userspace.
1196 Cross-Referencing Space Metadata
1200 cross-referencing records between metadata structures.
1240 1. Any space mentioned by any metadata structure are cross-referenced as
1335 Most typically these fragments are metadata about the file -- origins, security
1444 fixed-size metadata records -- each dabtree block contains a magic number, a
1490 Cross-referencing these values against the filesystem metadata should be a
1491 simple matter of walking the free space and inode metadata in each AG and the
1520 the metadata are temporarily inconsistent with each other, and rebuilding is
1523 Only online fsck has this requirement of total consistency of AG metadata, and
1551 created by other writer threads that resulted in false reports of metadata
1565 actual metadata updates to a fresh transaction.
1595 but before #2 is committed, a scan of the filesystem metadata would show
1596 inconsistent filesystem metadata because there would not appear to be any owner
1608 In other words, all per-AG metadata updates for an unmapped block must be
1744 1. Lock the resource(s) associated with the metadata being scrubbed.
1847 shadow copy of an ondisk metadata structure in memory and comparing the two
1849 For online repair to rebuild a metadata structure, it must compute the record
1987 In XFS, each type of indexed space metadata (free space, inodes, reference
2191 The chapter about :ref:`secondary metadata<secondary_metadata>` mentioned that
2192 checking and repairing of secondary metadata commonly requires coordination
2193 between a live metadata scan of the filesystem and writer threads that are
2194 updating that metadata.
2196 metadata updates from the filesystem into the data being collected by the scan.
2198 applying them before writing the new metadata to disk, but this leads to
2543 old metadata blocks:
2615 If there are, the space metadata inconsistencies are reason enough to abort the
2775 Reaping Old Metadata Blocks
2785 Offline repair rebuilds all space metadata after recording the usage of
2797 For space metadata, the process of finding extents to dispose of generally
2801 The space reservations used to create the new metadata can be used here if
2805 same ``XFS_RMAP_OWN_*`` number for the metadata that is being preserved.
2811 Repairs for file-based metadata such as extended attributes, directories,
2870 metadata structure being rebuilt, set the corresponding range in a bitmap.
3012 careful to access the ondisk metadata *only* when the ondisk metadata is so
3069 that should reflect the ondisk metadata, at least when the filesystem has been
3089 Although online fsck can read the filesystem metadata to compute the correct
3096 filesystem metadata to get an accurate reading and install it in the percpu
3163 Certain types of metadata can only be checked by walking every file in the
3281 Obviously, it is an absolute requirement that the inode metadata be consistent
3301 creation of the incore context for an existing file does not require metadata
3319 the ondisk metadata and freeing the inode.
3349 9. Inode ``ILOCK`` for file metadata updates.
3351 10. AG header buffer locks / Realtime metadata inode ILOCK.
3353 11. Realtime metadata buffer locks, if applicable.
3633 filesystem objects until the newly collected metadata reflect all filesystem
3765 walk the surviving ondisk metadata looking for replacement metadata records,
3772 decides a structure is corrupt, no other threads can access the metadata until
3773 the kernel finishes repairing and revalidating the metadata.
3789 scrub, generate reverse mappings for all AG metadata: inodes, btrees, CoW
3846 XFS stores a substantial amount of metadata in file forks: directories,
3860 Therefore, online repair of file-based metadata createas a temporary file in
3873 Swapping metadata extents with a temporary file requires the owner field of the
3893 | In the initial iteration of file metadata repair, the damaged metadata |
3960 access to file data are controlled via the IOLOCK, and access to file metadata
3991 metadata must replace the old.
4201 out of space failure once it starts dirtying metadata.
4219 Regular users are required to abide the quota limits, though metadata repairs
4220 may exceed quota to resolve inconsistent metadata elsewhere.
4222 Special Features for Swapping Metadata File Extents
4227 Metadata repairs must take extra steps to support these cases:
4266 To repair a metadata file, online repair proceeds as follows:
4458 A parent pointer is a piece of file metadata that enables a user to locate the
4463 reconstruction of filesystem space metadata.
4552 functions are not allowed to modify filesystem metadata.
4637 hook functions are not allowed to modify filesystem metadata.
4724 Free space metadata has not been ensured yet, so repair cannot yet use the
4727 4. At the start of phase 6, space metadata have been rebuilt.
4798 program, ``xfs_scrub``, that provide the ability to drive metadata checks and
4803 Checking Metadata
4809 In XFS, there are several groups of metadata dependencies:
4828 metadata indices of the allocation groups and the realtime volume.
4829 This includes quota and realtime metadata files.
4831 f. Inode records depends on consistency within the inode metadata indices.
4833 g. Realtime space metadata depend on the inode records and data forks of the
4834 realtime metadata inodes.
4836 h. The allocation group metadata indices (free space, inodes, reference count,
4838 between all the AG metadata btrees.
4843 Therefore, a metadata dependency graph is a convenient way to schedule checking
4884 metadata object of each inode.
4901 each metadata object of each inode.
4924 Problems reported in any other space metadata are deferred to phase 4.
4929 file's metadata are repaired immediately if all space metadata were validated
4940 Each repair item represents a single lockable object -- AGs, metadata files,
4961 iii. If the object no longer requires repairs, revalidate all metadata
4976 iii. If the object no longer requires repairs, revalidate all metadata
5012 If ``xfs_scrub`` succeeds in validating the filesystem metadata by the end of
5089 This scan after validation of all filesystem metadata (except for the summary
5102 mapping ioctl to map the recorded media errors back to metadata structures
5112 rebuilding of its metadata indices, and how filesystem users can interact with
5137 When XFS v5 came along with self-describing metadata, this old mechanism grew
5228 ondisk metadata, the operation cannot be cancelled cleanly, after which a QoS
5248 To clear all the metadata out of a portion of physical storage, clearspace
5251 Next, clearspace finds all metadata blocks in that region by way of
5253 This often results in the metadata being rebuilt somewhere that is not being
5312 the data and metadata at the end of the filesystem, and handing the freed space