LiquidFiles Documentation
LiquidFiles Documentation

Attachment Processing Information

This article provides some background information how Attachments are processed in LiquidFiles.

When a file is upload to LiquidFiles, some of the things that may happen will take a non-trivial amount of time and will be executed in the background. Typically, we don't want to let a browser wait for longer than 30s without a response so any task that can take longer than 30s we execute as a background job.

Attachment Processing Tasks (in order of processing)

File Assembly

When a file is uploaded to LiquidFiles, if it's above 100MB in size, it's recommended that you split the file into 100MB parts (100MB chunks). This will happen automatically when you use the web interface or any of the plugins, and something you have to do yourself if you use the API. When all the chunks have been uploaded LiquidFiles will reassemble the file. If for instance you have uploaded a 10GB file, there's going to be 100 x 100MB chunks. Reading and writing 10GB can depending on your disk speed take a bit of time.

Checksum calculation

If a file has been uploaded in chunks, LiquidFiles will calculate the SHA-256 and CRC32 checksum after the file has been assembled.

Virus Scanning

Virus Scanning is controlled in Admin → Configuration → Settings. If you choose to enable virus scanning (virus scanning is enabled on default) then virus scanning will be performed as a background tasks. Virus Scanning in general is one of if not the most resource intensive task in LiquidFiles.

ActionScript Scanning

You can add your own custom Attachment Upload ActionScript to perform your own custom attachment validation. If you enable ActionScript scanning, it will be performed as a background job.

Why does this matter?

When you use the LiquidFiles web interface, you can see that when a large file has been uploaded, there will be a spinner indicating that the file is still being processed, and at the end it will process a completed check mark. But, LiquidFiles will permit that you click send before the processing has been completed. The rationale here is that in the time when the message is being delivered to the recipient and before they opens it, whatever background processing that needs to be performed has had time to complete in that time.

When you develop your own API integration, it also important to know that the immediate response from the API call to upload the file cannot be trusted, in the sense that just because you get a success message from the API, that only means that the file was successfully uploaded. It does not yet say if a file is infected with a Virus, or might be blocked by an ActionScript as part of the post processing that happens after the file has been uploaded successfully.

The Quick solutions

You can poll the attachment you've uploaded using the Attachment Show API. When the processed boolean flag has been set to true, that means that all post processing has completed and you can check if the content_blocked flag is true or not. If processed == true and content_blocked == false that means that there will be nothing stopping the user from downloading the file (if they have the correct permissions of course).

Processing Details

Lets look further into the details of the post processing from the API, looking at a subsection of the Attachments Attributes:

Post Processing Response Parameters
Parameter Type Description
assembled Boolean True if the attachment has been assembled or not (when uploading multiple chunks)
virus_scan_required Boolean True if the attachment is required to be Virus Scanned.
virus_scanned Boolean True if the attachment has been Virus Scanned.
virus_scanned_at DateTime When the Virus Scanner last scanned this attachment.
content_blocked Boolean True if the attachment has been blocked (by the Virus Scanner or ActionScript).
content_blocked_message String If the attachment has been blocked, this will display the reason.
actionscript String If the attachment needs to be scanned by an ActionScript Upload script, this will list the name of that script.
actionscript_scanned Boolean True if the ActionScript scan has been performed.
actionscript_scanned_at DateTime When the ActionScript was executed.
processed Boolean True when the attachment has completed all required Assembly, Checksum, Virus and ActionScript scanning.
processed_at DateTime When the Attachment completed all required processing.
available Boolean True if the file is available for download or not. Please note that this is user context sensitive. If a file is required to be virus scanned and you have a setting in Admin → Configuration → Settings that Local Users can download files that are not virus scanned, then if the current user is a local user, this available will be true regardless if the virus_scan has completed or not. For external users or if Local Users are not permitted to download files that haven't been virus scanned, this will be false unless the virus scan has completed with no virus detected.

You can see assembled, virus_scanned and actionscript parameters as well as processed, available and content_blocked parameters more meta parameters.

The assembled flag means in the strict context that the file has been assembled and the checksum has been calculated. Before an attachment has been assembled, it's not accessible regardless of other settings. When the attachment is assembled, it will be accessible for Admins with permission to access user data to download the files.

The virus_scanned and actionscript_scanned parameters will be set if you've enabled Virus Scanning (Admin → Configuration → Settings) and actionscript will be set if you have enbled Attachment Upload ActionScript scanning.

The processed flag is set when all required post processing has completed. So depending on if you've enabled virus scanning and Actionscript scanning, and if the file was uploaded in chunks or not, when everything has completed, processed will be set to true.

Available, as outlined in the table above, is user context sensitive, so an Admin with access to user data will have an attachment available == true as soon as the file has been assembled, where's an external user will only have available == true when processed == true and content_blocked == false.