How to store extended file attributes in iCloud

Apple's macOS and some other operating systems store descriptive information about files known as metadata on storage volumes in extended attributes (xattr). Extended attributes contain additional information about files used by operating systems.

By storing and retrieving extended attributes, operating systems can modify info about files without changing file contents directly.

For example in macOS Finder, when you select a file and press Command-I on the keyboard, Finder opens a Get Info window and reads some of the file's extended attributes and displays them.

Some extended attributes are created and managed by macOS itself, but applications can create their own extended attributes and store them as well. It is up to each app to decide how it wants to manage extended attributes.

Some filesystems don't support extended attributes at all, such as NFS. If you copy files to a filesystem which doesn't support extended attributes, those attributes are lost forever.

Most common modern filesystems do support extended attributes, but may manage the data in different ways. On most operating systems, extended attributes are not stored with files themselves, but in volume metadata catalogs, which stores the additional attributes in lists or databases, for each file.

When a file gets copied, the metadata gets copied along with it and inserted into the destination volume's metadata catalogs to preserve the additional attributes. Using this behavior operating systems move file metadata from one volume to another along with files themselves during copies.

Apple publishes limited, but not all, developer info about the macOS metadata system and how it works. There currently is no user-level Apple documentation for managing metadata on macOS or iOS.

xattrs and reverse domain name organization

We do know that in macOS, extended attributes and metadata are organized under reverse domain notation - much like app bundle names and company organization are. A few Apple reverse domains used for metadata include:

1. com.apple.FinderInfo
2. com.apple.metadata
3. com.apple.TextEncoding

All extended attributes are not required in every file - some files may be missing xattrs from certain reverse domains entirely.

xattrs in iCloud

When Apple released iCloud Drive as a way of copying and storing files in the cloud, users began to notice that some, but not all file metadata was lost. Worse, this behavior became inconsistent across different versions of macOS, sometime around the release of Mac OS X Mojave.

Users noticed that some — but not all — extended attributes get stripped from files when moved into iCloud and then copied back out.

At minimum we know that all extended attributes from the com.apple.FinderInfo and all user-added metadata including Get Info comments in the com.apple.metadata domain get deleted when files are moved into iCloud.

Extended attributes such as those in the com.apple.TextEncoding domain and certain other domains are preserved. In a way this makes sense since every app and even the Finder must know how text data inside each file is encoded in order to display it correctly.

On the other hand it may not make sense to Apple for iCloud to preserve some attributes such as com.apple.FinderInfo since that data may be unique to Finder display preferences on each user's Mac.

Still, stripping files' Get Info user comments is clearly data loss. Worse, there have been reports of iCloud Drive stripping third-party extended attributes, from files across different versions of macOS when accessing the same files is in iCloud. This is clearly bad.

So far, Apple hasn't publicly addressed or commented on this, so we don't know if this is intended behavior or an unaddressed issue.

A janky fix (sort of)

What Apple has done since sometime around 2013 is to quietly add sort of a developer hack to file metadata called extended attribute tags.

Apple's xattrs tags system consists of appending one of four characters (C, P, N, or S) to the end of each xattr preceded by the #' character. Since #' is one of the characters forbidden in xattr names, flags gets ignored by most application code that handles xattrs.

Each of the four flag characters corresponds to an xattr metadata preservation level in Apple's programming interfaces. We won't get into those details here, but suffice it to say the P, N, and S flags correspond, roughly to these attribute preservation levels:

1. C - Preserve when file data changes.
2. P - Preserve during copying.
3. N - Always strip xattrs - even when copying.
4. S - Always preserve xattrs when syncing.

Hence, when copying files into iCloud Drive, any file xattrs which have the "#S" flag appended to the end will be preserved.

Currently there is no easy way to make these modifications to files without using the Terminal and understanding exactly what changes should be made.

And we should warn you that altering extended attributes can be risky business. Any errors in xattrs can change the way files are displayed in Finder, Get Info windows, file creation and modification dates, and can even in some cases render files damaged or unusable.

We know for certain that as of macOS Ventura, Spotlight indexing knows how to handle additional xattr flags without confusion or data loss, but the Finder's Get Info windows don't always. Usually, adding xattr flags to xattrs which the Finder's Get Info uses can render files unreadable and possibly even unopenable from the Finder.

Also note that the Finder will always honor any xattr flags during copies - for example if the '#N' flag exists in any xattrs in any file during a local Finder copy, those attributes will always be stripped from the copied files in all cases.

On the other hand, the Terminal file copy command (cp) will always preserve all xattr flags in all cases - copying all xattrs on all copied files no matter what.

You can view the rather sparse developer info on xattr flags in Terminal by typing:

man xattr_name_with_flags and pressing Return.

But unless you're a developer, the info provided may be difficult to understand since it's mostly Standard C Library programming info included in the C header file xattr_flags.h

xattrs in Terminal

You can view and alter files' xattrs in the macOS Terminal using the xattr command, but be extremely careful when doing so.

You can also use standard Terminal directory listings to see whether a given file has xattrs or not. To do so, in Terminal, navigate to the directory whose files you want to view using the 'cd' command, then type:

ls -l

and press Return.

You'll see a listing of the directory, similar to this:

On the left side of the directory listing are each file's UNIX permissions. File permissions which end in "@" or "+" indicate files that have extended attributes attached to them.

To see which reverse domains each file's attributes belong to, type:

ls -l@

and press Return.

The listing will be the same as using "ls -l", except that any xattr reverse domains used in files will also be listed:

The xattr UNIX command is a way to view and change xattrs on files, but as we mentioned above, be extremely careful when using it. The command is powerful and can destroy file usability easily.

You can view all xattr command options in Terminal by typing:

man xattr and pressing Return.

Press Control-Z to exit the man system.

In summary, there isn't any simple and easy way to deal with xattrs when copying files to and from iCloud.

As long as a given xattr's flag is set to '#S' when syncing to or from iCloud, that xattr will always be preserved.

Apple should fix this in the macOS Finder and in iOS, but doing so presents a problem: it would require adding the '#S' tag to all file xattrs when syncing, and doing so would likely break at least some third party apps which read and write xattrs but don't yet know how to handle tags.

xattr resources

Developer Howard Oakley has written several apps and tools to help deal with xattrs - including an editor called xattred (rhymes with 'scattered'), and the command line tool cmpxat which compares and displays all xattr differences between two files.

He's also written Cirrus - an iCloud browser tool - and his blog has several discussions on the iCloud xattr issue.

Looking directly at iCloud's files and folders with Cirrus1.0b4

He also has two very interesting virtualization apps, Viable and Liviable which run macOS and Linux virtual machines on Apple Silicon Macs.

Michael Tsai has a brief page on xattr Flags and iCloud Drive with a similar discussion.

Apple has a File Metadata Attributes Reference but it only covers Spotlight, iCloud, and Core Data attributes.

If you're a developer, the public Swift or Objective-C Apple file metadata API documentation is also available.

If you can read C code, Apple's open source file xattr_flags.c shows you the inner workings of the xattr tag code Apple added in 2013.

There is an older technical discussion of how xattrs are saved on StackExchange but it's rather technical and delves into the inner workings of the BSD filesystem - and it's more specific to the Linux ext3 filesystem than macOS.

Apple's File System Programming Guide is for developers, but again, it's extremely technical and lengthy - and is more general.

Hopefully in the near future Apple will arrive at a solution which solves all of the above iCloud problems and seamlessly preserves xattr metadata across all iCloud file syncing.