• XFS was more fragile, but the issue seems to be fixed.
• XFS was surely a slow-FS on metadata operations, but it has been fixed recently as well.
• EXT4 is still getting quite critical fixes as it follows from commits at kernel.org's git.
• "EXT4 does not support concurrent writes, XFS does"
• (But) EXT4 is more "mainline"

So, the final answer depends on your precise requirements (as usual).

A previous XFS Q/A mentioned this page: How to Choose Your Red Hat Enterprise Linux File System. I think it will address your question about performance :-). This is from the official Red Hat Knowledgebase. It would be inappropriate to copy the whole page. Here is a taste:

Another way to characterize this is that the Ext4 file system variants tend to perform better on systems that have limited I/O capability. Ext3 and Ext4 perform better on limited bandwidth (< 200MB/s) and up to ~1,000 IOPS capability. For anything with higher capability, XFS tends to be faster. XFS also consumes about twice the CPU-per-metadata operation compared to Ext3 and Ext4, so if you have a CPU-bound workload with little concurrency, then the Ext3 or Ext4 variants will be faster. In general, Ext3 or Ext4 is better if an application uses a single read/write thread and small files, while XFS shines when an application uses multiple read/write threads and bigger files.

Previous answer

Here is a previous version of this answer. I have adapted and expanded it, to provide context and corroborate the above source. It is not all essential reading.

It does not make sense to avoid XFS for being old. There are many similarities between XFS and ext2/ext3/ext4, including their long and active lives.

XFS is the main supported filesystem for RedHat Enterprise Linux. A lot of effort has gone in to develop XFS over time, particularly from RHEL. The sheer number of new and advanced features is higher in XFS :-). See e.g. support for "reflink".

"Desktop Linux" installs - which also target modern laptops - still generally use ext4. ext4 provides a very familiar, well-known environment.

On an average desktop PC, the advantages of XFS do not necessarily apply. ext4 might have advantages, including performance advantages. Although if you take the figures above at face value, they suggest XFS might have an advantage on fast modern laptops!

Currently, the ext4 maintainer is employed by Google. ext4 is used in Google's Android OS. So ext4 is used in many phones and tablets, which have less powerful hardware.[1][2]

I also find ext4 convenient because you can shrink the filesystem partitions. This is potentially useful when you want to manage the space used by different installs on a single disk. XFS does not have support for shrinking, only growing. (In principle, this can be addressed within LVM, using "thin provisioning". See Stratis Storage, which uses XFS together with thin provisioning. Perhaps in future Stratis will take off, and all distros will be installable to Stratis).

In some cases, the ext4 design has been described as old and messy. XFS clearly managed a bit better here: Simple file copy (or write) causes ten second+ latency on Linux filesystem. UPDATE: this link now mentions patches which have been submitted to improve ext4 here, probably for v5.3 of the Linux kernel. And there could still be significant improvements in future.

XFS "small file performance"

XFS performance when dealing with many small files was originally a major weakness, relative to other filesystems. Improvements have been made over time.

A big change to be aware of is delayed logging, sometime around 2010-2012. I.e. in case you see any small-file performance comparisons which pre-date this feature, they are no longer relevant. The delayed logging design was heavily based on ext3 logging :-). Nevertheless, there were still performance differences between the two in different cases. See: https://lwn.net/Articles/476263/

(Before that, apparently "the version 2 log format was added in 2002, which helped make metadata performance much better". But this probably does not help answer this question :-).

There are some further changes since then. E.g. the Red Hat document might not reflect all the most recent changes. I noticed this quote:

[...] free inode B+ trees (finobt) for better performance on "aged" filesystems, and file types are now stored in the directory by default (ftype=1) which greatly improves performance in certain scenarios with lots of (usually small) files.

-- comment on What is the most high-performance Linux filesystem for storing a lot of small files (HDD, not SSD)?

You can use ext4 but I would recommend mounting with journal_data mode which will turn off dealloc (delayed allocation) which 'caused some earlier problems. The disabling of dealloc will make new data writes slower, but make writes in the event of power failure less likely to have loss. I should also mention that you can disable dealloc without using journal_data which has some other benefits (or at least it did in ext3), such as slightly improved reads, and I believe better recovery.

Extents will still help with fragmentation. Extents make delete's of large files much faster than ext3, a delete of any sized data (single file) should be near instantaneous on ext4 but can take a long time on ext3. (any extent based FS has this advantage)

ext4 also fsck 's faster than ext3.

One last note, there were bugfixes in ext4 up to like 2.6.31? I would basically make sure you aren't running a kernel pre 2.6.32 which is an LTS kernel.

Hmmm. How do I detect a failed XFS filesystem?

I've been using XFS for ages. But... I guess I don't detect it, at all. If it mounts, I trust it works. That's how most people do it... filesystem checks are automated, if it boots and it's up and running, that's that.

Now, don't get me wrong. I actually do a ton of monitoring, but none of it is filesystem specific. I run SMART selftests (using select,cont to do a disk segment per day, because long simply takes too long). I run RAID checks (also in segments) and also check that there are no mismatches in parity (mismatch_cnt = 0). I get instant mail notification if any of those fail and I actually replace HDDs once they start reallocating sectors (or at least, no longer trust them with important data).

So I have monitoring to make sure the storage works as it should. This covers errors inside the drives themselves (SMART) and to some extent also errors on a higher level (RAID checks in a way also test controllers, cables, RAID logic, ...).

As long as that works fine, the filesystem better be fine, too. Outside of checksumming filesystems like ZFS/btrfs (maybe XFS in the future, too) it's not really part of the concept to run checks on a filesystem level while mounted, apart from whatever sanity checks the filesystem itself does internally.

Your output suggests you're running RAID too, and had a failed disk in that RAID; even so that really should not cause errors happening on md0, unless it was RAID without redundancy (RAID0 or already degraded RAID1/5/6/10).

You should fix your problems below the filesystem layer first. You can hardly blame XFS for disk errors and that's not how you check for disk errors.

I guess if you really wanted to run a full read test on top of the filesystem, you could do an xfsdump to a backup disk... if you're doing a full read test of your filesystem anyhow, might as well do it in a way that's meaningful somehow.

It's the nature of xfsdump to walk the XFS filesystem in its entirety and store all the files. So that should come as close as possible to a full read test, not including free space.

Of course, if you're already running another backup system, that's really the same story in a filesystem-agnostic way (and if that backup system encounters read errors that aren't just lack of permissions, it better send a mail report to you, too), although of course if it's an incremental backup, without periodic full backups it won't actually read a file more than once...

But in general, we trust filesystems to "just work" as long as the storage is known to work. While it would be nice to have each and every program without exception to elevate any and all I/O errors it encounters, I'm not aware of a generic purpose solution to actually do so. Each program does its own error handling.

I think you can do this by disabling one of the features that Rhel8 provides. Format the space on Rhel8 with mkfs.xfs -m reflink=0 /dev/sdN The -m reflink=0 will allow mount on both systems but the relink feature on Rhel8 will not be usable. Also see https://access.redhat.com/solutions/4582401 (will need Redhat login) for more details.

To quote Archimedes:

Eureka! I've found it!

The problem is that when you invoke the command xfs_admin -U generate /dev/sdXY the generate parameter causes xfs_admin to flip the incompatible bit to on, so that older kernels cannot use the filesystem. An explanation can be found in the following quote provided by RTFM:

xfs_admin(8)

...

-U uuid
          Set the UUID of the filesystem to uuid.  A sample UUID looks
          like this: "c1b9d5a2-f162-11cf-9ece-0020afc76f16".  The uuid
          may also be nil, which will set the filesystem UUID to the
          null UUID.  The uuid may also be generate, which will generate
          a new UUID for the filesystem.  Note that on CRC-enabled
          filesystems, this will set an incompatible flag such that
          older kernels will not be able to mount the filesystem.  To
          remove this incompatible flag, use restore, which will restore
          the original UUID and remove the incompatible feature flag as
          needed.

To reverse this, simply run xfs_admin -U restore /dev/sdXY, and you should be able to mount the partition again. What's strange is that the machine that was used to generate the UUID is the machine that's running the latest kernel (4.2.5-1-ARCH, okay, maybe not the latest, but UUID should imply that this machine is not excluded from compatibility). Anyway what's important is that the problem was solved.

It seems that is the relink feature flag, and it would make sense that such a file system could only be mounted read only on a kernel predating this (IMHO very useful) file system feature.

There's no tool to disable that feature after it's been used at file system creation time. You will have to make a new file system and copy over the contents of the old.

mkfs.xfs -m reflink=0 -f yourfilesystemimage

why don't you try? fallocate -l 2G img; mkfs.xfs img; sudo mount -o loop img /mnt; fallocate -l 4G img; sudo xfs_growfs /mnt should do that and give you an instant answer.

Anyways, no, growing modern filesystems, and especially XFS, is not a heavy operation. There's no data to be moved – there's just new free space; a couple of data structures need to be updated / created (if the new size implies more allocation groups). All in all, nearly no work.

Update

After checking xfs_fsr's source code I found this section:

/* Check if the temporary file has fewer extents */
new_nextents = getnextents(tfd);
if (dflag)
    fsrprintf(_("Temporary file has %d extents (%d in original)\n"), new_nextents, cur_nextents);
if (cur_nextents <= new_nextents) {
    if (vflag)
        fsrprintf(_("No improvement will be made (skipping): %s\n"), fname);
    free(fbuf);
    close(tfd);
    return 1; /* no change/no error */
}

I re-run the command with the -d (debug flag) and it returned the following:

xfs_fsr /dev/md1 -v -d
/mnt/disk1 start inode=0
ino=133
ino=133 extents=4 can_save=3 tmp=/mnt/disk1/.fsr/ag0/tmp23917
DEBUG: fsize=30364684107 blsz_dio=16773120 d_min=512 d_max=2147483136 pgsz=4096
Temporary file has 4 extents (4 in original)
No improvement will be made (skipping): ino=133

This means the original file consists of 4 file parts across the HDD platters and creating a new defragmented file would still consist of 4 file parts so this file has been skipped. But why isn't it able to merge them? I don't know so I asked at kernel.org bugzilla.

Update 2

I received an answer. As my XFS filesystem (and probably yours) has a block size of 4KB, the maximum size of a file part can not be bigger than 8GB as stated in the XFS docs:

If a file is zero bytes long, it will have no extents, di_nblocks and di_nexents will be zero. Any file with data will have at least one extent, and each extent can use from 1 to over 2 million blocks (221) on the filesystem. For a default 4KB block size filesystem, a single extent can be up to 8GB in length.

So "No improvement will be made" means "No improvement possible" ;)

Original Answer

I'm having the same problem and found out that there are many cases where xfs_fsr skips the defragmentation:

 * mandatory locks are present
 * file is marked immutable, append-only, or nodefrag
 * filesystem is shut down
 * change/modify times have been altered since defrag started
 * original file is mmapped

If you use -v, xfs_fsr returns the inode numbers of the files that where skipped:

xfs_fsr /dev/md1 -v
/mnt/disk1 start inode=0
ino=133
No improvement will be made (skipping): ino=133
ino=135
No improvement will be made (skipping): ino=135
....

With this inode number you are able to check the attributes of the file with the bmap -a command of xfs_db:

xfs_db -r /dev/md3
xfs_db> inode 133
xfs_db> bmap -a
xfs_db> bmap -d
data offset 0 startblock 1314074773 (4/240332949) count 2097151 flag 0
data offset 2097151 startblock 1316171924 (4/242430100) count 2097151 flag 0
data offset 4194302 startblock 1318269075 (4/244527251) count 2097151 flag 0
data offset 6291453 startblock 1320366226 (4/246624402) count 1121800 flag 0
xfs_db> quit

As you can see it returns in my case an empty value as there are no attribute flags present.

So I tried to check as next thing the lock status. At first we obtain the file name of the inode as follows:

find /mnt/disk1 -xdev -inum 133
/mnt/disk1/foo/bar.dat

And then we try to get the lock status:

lsof /mnt/disk1/foo/bar.dat

As the result is empty, nothing seems to lock this file.

Now lets check if the file has been changed since the last defrag try:

ls -l /mnt/disk1/foo/bar.dat
-rw-rw-rw- 1 nobody users 30364684107 Nov 18  2019 /mnt/disk1/foo/bar.dat

Nope, looks still "old".

Bad Things(TM) may happen if a filesystem is 100% full (or perhaps 105% on account of the root reserved space). Specific cases I have seen (though these are not for XFS):

• 100% full NTFS partition (we had warned the user not to run their laptop that full but they did for months...) ended up with filesystem loops so when you tried to backup data off of that 300G disk you could fill a 3T disk or whatever because loops.
• Some linux virtual machines with near 100% full disks due to a bug began writing end-of-partition bits back at the beginning of the partition, thus destroying various important bits that happen to live there.

So yes if there is a bug you could end up with a corrupt filesystem; as a precaution on test systems or during bake-in one might completely fill the disk to see what happens. Usually however the partition just fills up and writes fail without any OS or filesystem corruption (of course application software may then be Very Unhappy(TM) as the programmers never tested what happens when writes fail... e.g. stunnel stopped processing application data when it could not write logs)

(There are legitimate cases where a filesystem might be run 100% full, e.g. a database might want all the space, or it's a (mostly?) read-only partition with scads of genome data, etc.)

Otherwise there are typically no protections in place; usually some sort of monitoring gets setup that warns when a partition is X% full, or shows a large change in space consumed. Often these things can be predicted in advance based on growth over time: it's better to send an email on say Thursday ("partition X will hit 90% in three days...") than an emergency page at 3AM on a Sunday (90% full - error! error! error!).

From the documentation you cited:

The XFS filesystem [...] does not require any change in the default settings, either at filesystem creation time or at mount.

Source: https://kafka.apache.org/documentation/#xfs

So it should just work. Also there is nothing special anymore about a 20TB device size.

Consider adding a partition table and then use /dev/sdb1 instead of /dev/sdb.

So, does it matter in my case, or my situation is too ordinary to justify a strong preference for a file system in particular?

The latter. The way you describe this workload, I think it is not very demanding. Both ext4 and XFS should be able to handle it. So I think you should have no strong preference, except to consider what you are familiar with and what is best documented.

If you use Debian, Ubuntu, or Fedora Workstation, the installer defaults to ext4. So that's what most Linux users would be familiar with.

This can make differences as there are a few functional differences, i.e. a few little traps that might catch you out if you are used to the other one.

XFS is a great filesystem, that scales well for large servers. But in one case it has avoided meeting the same expectation, that AIUI Linus required ext4 to fulfil as the successor to ext3. This is an example of where sticking to what everyone else uses for whatever purpose, might help you avoid hitting things that nobody else knows to warn about :-). Which filesystems require fsync() for crash-safety when replacing an existing file with rename()?

Red Hat is trying to grow a storage stack based on XFS called Stratis, including specific features and/or new work that will go into XFS. This might become interesting and lead to more widespread community expertise etc with XFS in future. And if you want to use RHEL or CentOS somewhere, which default to XFS, then by all means go ahead. Red Hat make a lot of comprehensive manuals available (which basically apply to CentOS too).

Our glorious future of checksumming filesystems seems equally cloudy at the moment ... at least Red Hat starting Stratis suggests someone sees it that way ... so if you don't mention them, then I will also try to avoid doing so. You should require your backup application to include checksums anyway. Even if it doesn't give you the full goodness of ZFS' RAID support.

My main concern is that such damage NEVER happened to disks partitioned to NTFS, whatsoever.

It may have never happened to you, but it has happened. The only filesystems that can claim things like this never happening are those that have never been exposed to such conditions. Even BTRFS and ZFS, which are both designed to be resilient against stuff like this, can have such issues.

To your actual questions though:

Is ext4 safe in general? I mean is it me or are there other people who experienced loss of disks/information on disks formatted to etx4?

It depends on what you mean by 'safe'. I've personally lost data on disks formatted with ext4, but every time it's happened to me it's been due to bad hardware, and, more importantly, it would have happened eventually with pretty much any other filesystem. Despite this, I do still use it for numerous things on a regular basis because, barring user error or hardware issues (which includes unexpected power loss), it just works. So, I consider it 'safe' by most people's definitions, but you may or may not.

In Linux world, what could be a safer alternative to ext4 that can outlive unexpected electricity switch off or unexpected unmounting?

No, not unless you want to deal with other limitations or issues. In particular:

• XFS is a bit more resilient against unexpected power loss and doesn't need long checks on reboot like ext4 does, but has a number of practical limitations that make it questionable for small-scale use (can't shrink filesystems, performance isn't quite as good as ext4 on a new volume, can't do data journaling).
• NILFS2 is almost impossible to kill with a power failure, but you might lose 30 or so seconds of changes, it requires a userspace component when mounting, and it is missing a handful of features that are generally considered standard by most Linux filesystems.
• BTRFS will save you from failing hardware and reasonably reliably, plus it provides nice support for online replacement of failing disks, but again you may lose some of the most recent changes on an unexpected power loss, and you need to do a lot more to keep the volume healthy than for most other filesystems.
• ZFS has all the benefits that BTRFS does with none of it's issues (except the management ones), but it requires you build a third-party kernel module and you won't get get any upstream support for any issues you have if you're not running on enterprise grade hardware.

You can, however, do a number of things to make ext4 safer:

• Change the behavior when errors are encountered. By default, if an error is encountered in filesystem metadata, ext4 will just mark the volume as needing to be checked, and then act like nothing happened. It's the only filesystem on Linux that does this, everything else will remount the volume read-only, thus preventing any writes to the filesystem from making things worse. You can get this behavior on ext4 by adding errors=remount-ro to the mount options, or running tune2fs -e remount-ro on the block device containing the filesystem.
• Make sure you're not using writeback mode for the journal. Yo can ensure this by double checking the mount options for the volume and making sure that journal=writeback is not in the list. Journal writeback mode can significantly improve the performance of certain workloads on ext4 filesystems, but it makes it much more likely that you lose data if you unexpectedly lose power.
• If you want to be really paranoid about data safety, you can enable journaled data mode. Normally, the journal on an ext4 filesystem only tracks changes to metadata (renames, file deletion or creation, timestamp updates, etc). In journaled data mode, all changes go through the journal. This slows things down significantly, but provides a functionally 100% guarantee that the file system will remain internally consistent. You can enable this by passing journal=data in the mount options.
• You can add the auto_da_alloc mount option. Essentially, this detects applications not calling fsync() when they should, and properly handles things. It's not the default because it slows things down a bit, and most applications don't need it.
• On newer kernels, you can enable journal checksumming. This won't actually 'save' your data, but it will help ensure that you're not getting bogus data back when there was an error. This can be enabled by adding journal_checksum to the mount options.
• If you've got a new enough kernel and version of e2fsprogs, you can enable metadata checksumming. Similar to the journal checksumming, this won't save your data, but it will help prevent you from seeing bogus data if there's an error. This has to be enabled at filesystem creation time, by passing -O metadata_checksum,metadata_checksum_seed to mkfs.ext4. If you do this, you (probably) don't need to also enable journal checksumming, as the journal is part of what gets covered by the metadata checksumming.

Because nobody wrote that yet :-)

Some related links explaining what needs to be done:

• https://xfs.org/index.php/Shrinking_Support
• https://lwn.net/Articles/476808/
• https://marc.info/?l=linux-xfs&m=118091640624488&

And also:

The reason it hasn't been done is that there is basically no demand for shrinking large filesystems. Storage is -cheap-, and in most environments data sets and capacity only grow.

I personally don't 100 % agree with that but it makes sense. I guess the idea behind XFS wasn't to be used on workstations or generally situations where shrinking is necessary so it was written in a way that shrinking is harder (with transaction log in the middle of the filesystem etc.) and growing easier.

I think XFS developers are trying to say: If you want shrinkable filesystem, simply don't use XFS, but patches welcome :-)

Update: Support for shrinking is currently being worked on. xfsprogs 5.12 released in May already supports shrinking but it's still experimental -- you need unused space at the end of the filesystem and there isn't option to "defragment" the filesystem to make sure all unused space is at the end. But it's definitely moving forward and hopefully we can expect "full" support soon.

No, just editing /etc/fstab cannot cause xfs_repair to be executed.

For other filesystem types, it would work. But XFS is special here.

Changing the 6th field of /etc/fstab to a non-zero value on a XFS filesystem will cause the system to run fsck.xfs, whose man page says:

NAME
       fsck.xfs - do nothing, successfully

[...]

       However,  the  system  administrator  can force fsck.xfs to run xfs_re‐
       pair(8) at boot time by creating a /forcefsck file or booting the  sys‐
       tem with "fsck.mode=force" on the kernel command line.

So, ordinarily fsck.xfs will do nothing at all.

If you really want xfs_repair to run at boot, there are two conditions that both must be satisfied:

a) The 6th field of /etc/fstab must be non-zero for the XFS filesystem in question, so that fsck.xfs will be executed.

b) Either a /forcefsck file must exist on the root filesystem (or perhaps within initramfs, if planning to check the root filesystem), or the kernel command line must have the fsck.mode=force boot option. This will cause fsck.xfs to run xfs_repair instead of doing nothing.

What's so special with xfs_repair, then?

The XFS filesystem and the xfs_repair tool both will assume that the underlying disk is in good condition, or at least is capable of transparently replacing bad blocks with built-in spare blocks (as all modern disks do). If a modern disk has persistent bad blocks visible to the operating system, it usually means that the built-in spare block mechanism has already been overwhelmed by the amount of bad blocks, and the disk is probably going to fail completely soon enough anyway.

The man page of xfs_repair says:

   Disk Errors
       xfs_repair aborts on most disk I/O errors. Therefore, if you are trying
       to  repair  a  filesystem that was damaged due to a disk drive failure,
       steps should be taken to ensure that all blocks in the  filesystem  are
       readable and writable before attempting to use xfs_repair to repair the
       filesystem. A possible method is using dd(8) to copy the  data  onto  a
       good disk.

So, you probably should not set xfs_repair to run automatically in normal circumstances.

If a XFS filesystem has errors, you should always first evaluate the condition of the underlying disk: smartctl -a /dev/<disk device> might be useful, as might be using dd to read the whole contents of the partition/LV to /dev/null and seeing that the command can complete without errors.

If the disk is failing, you should first copy the contents of the partition/LV to a new, error-free disk (perhaps using dd or ddrescue), and only then you should attempt to run xfs_repair on the filesystem on the error-free disk.

Running xfs_repair automatically at boot time might be an appropriate workaround if you know that something is causing filesystem-level errors even when your disks are in good condition. But that is just a workaround, not a fix: you should find out what is causing the filesystem errors, and fix the root cause. (Maybe a filesystem driver bug, requiring an updated kernel package to fix?)

Thanks to responses in the comments, I was able to identify the problem as XFS filesystem related.

Essentially, the filesystem was running out free space for creating inodes. The XFS filesystem seems to be using the first one Terabyte of the disk to do so.

You can at least find out what's wrong by running:

xfs_repair -n /dev/mapper/rhel-root

-n runs xfs_repair in no-modify mode.

If it's complaining about it being mounted and writeable, you might want to try re-mounting it read-only (mount -r -o remount <device>), but this will probably just come back with '/dev/mapper/rhel-root is busy'.

I'd go with booting from a different medium and running xfs_check from there.