Upload File Shares API

Before uploading our first file, a little bit a background information is required. To upload the actual file to the LiquidFiles system, we have three different methods — Binary Uploads, HTML Form Based Uploads or JSON based uploads.

Upload Method Overview

Binary Uploads

This is the method that's used by all the builtin LiquidFiles functions. So if you upload a file using the web interface, the Outlook plugin, Mac plugin and so on, it's using the binary upload method. It's basically sending the raw binary data to the LiquidFiles system and is the most efficient way of uploading files to a web server. It is also probably the most complex/least supported in terms of support from external libraries if you want to incorporate this into an existing project.

JSON Based Uploads

The typical modern web based API format is JSON, which is a text based format that is also used by LiquidFiles for all LiquidFiles API functions. It's a simple, well documented format that is strictly text based, which means that we have to Base64 encode binary data (which basically stretches 3 bytes of binary data over 4 bytes only using text characters) that is a lot less efficient than either the binary or the HTML based upload methods, but has the benifit of being able to be easily defined in API. If you're sending smaller files (sensitive PDF documents that are fairly small for instance), this is the easiest to use.

Binary Data Uploads

There's two parts to a HTML POST statement, it's the body of the POST and it's the parameters. In a traditional HTML post, the parameters are hidden within the body of the POST, but there's no reason they have to. With the LiquidFiles binary upload method, we're actually using GET style parameters to pass additional information. It looks like this:

POST https://liquidfiles.example.com/shares/someshare/folders/somefolder/files/upload?name=somefile.ext
RAW BINARY FILEDATA

Example Using curl

The Response has been formatted for readability (added indendation and linebreaks).

curl -X POST --user "DaDMcRpYEFUsxSUjUozMoU:x" \
     -H "Accept: application/json" \
     -H "Content-Type: image/gif" \
     --data-binary "@someimage.gif" \
     https://liquidfiles.example.com/shares/project-1/folders/folder-1/files/upload?name=someimage.gif

{"file":
  {
    "id":"someimage-gif",
    "folder_id":"folder-1",
    "name":"someimage.gif",
    "size":49656,
    "size_to_human":"48 KB",
    "content_type":"image/gif",
    "checksum":"a5f21ee04ff2ff7ac191a76bf5bf432ac32eda5f2599bb2e876568d1376a0984",
    "crc32":"9d378437",
    "av_scanned":false,
    "av_infected":false,
    "deleted":false,
    "created_at":"2021-04-28T22:35:08.983Z",
    "updated_at":"2021-04-28T22:35:08.983Z",
    "processed":false,
    "processed_at":null
  }
}

Checking Individual Chunks before each upload

This is the same method that's described in the Attachments Chunks documentation. Before a chunk is uploaded, we can query the server and if we get a "partial content" response (HTTP reponse code 206) that means that the chunk doesn't exist and we can go ahead and upload it. If the server response with an "ok" message (HTTP response code 200) that means that the chunk already exist and we can move to the next one.

Example Request using curl

The following example will check if the first chunk (chunk 0) already exist on the system. If the reponse is a 206 response the chunk doesn't exist, if the response is 200, the chunk already exists.

curl -w "%{http_code}\n" --user "Y9fdTmZdv0THButt5ZONIY:x" \
     -H "Accept: application/json" \
     -H "Content-Type: application/zip" \
     https://liquidfiles.company.com/shares/test-share/folders/test-folder/files/upload?name=bigfile.zip&chunks=10&chunk=0

206

Or a more complete (bash) example how this would actually be used:

if [[ `curl -w "%{http_code}\n" --user "ojubiigauS2TxTy4ovk6Ph:x" \
     -H "Accept: application/json" \
     -H "Content-Type: application/zip" \
     https://liquidfiles.company.com/shares/test-share/folders/test-folder/files/upload?ame=bigfile.zip&chunks=10&chunk=0` -eq 206 ]]; then

  curl -X POST --user "ojubiigauS2TxTy4ovk6Ph:x" \
       -H "Accept: application/json" \
       -H "Content-Type: application/zip" \
       --data-binary "@bigfile.zip.00" \
       https://liquidfiles.company.com/shares/test-share/folders/test-folder/files/uploadd?ame=bigfile.zip&chunks=10&chunk=0
fi

So we first do a GET request for the filename, chunks and chunk to see if the chunk exist and if it doesn't we upload it.

Available Chunks

As an alternative to the above, you can also check all chunks that are available for a given file. Uploaded chunks they are stored for 5 days before being removed (if the file was not uploaded completely). One thing to note is that a chunk (or a file for that matter) is interrupted during transit, the partially uploaded chunk (or file) will be discarded. If you want to resume uploads, you can query the server using this API call to see what chunks are available and upload any missing chunks.

Request

Example Request using curl

#!/bin/sh

# Some nice variables
api_key="Y9fdTmZdv0THButt5ZONIY"
server="https://liquidfiles.company.com"

curl -X GET --user "$api_key:x" -H "Accept: application/json" \
  $server//shares/project-alpha/folders/root/files/available_chunks?name=Presentation1.pptx

{"chunks":[
  {
    "id":0,
    "size":104857600
  }, {
    "id":1,
    "size":104857600
  }]
}

In this case the first 2 chunks have been uploaded, both with 100Mb size. You can now continue with chunk 3 (chunk id 2, when starting from 0) starting at 200Mb.

JSON based uploads

JSON is what's being used for all other aspects of the LiquidFiles API but when it comes to sending files it poses a bit of a problem. Consider the following example request:

{"file":
  {
    "name":"logo.gif",
    "data":?????
  }
}

Since JSON is a text based protocol, we can't just insert binary data in it in the "data" tag. The standard programatical solution to problems like this is to Base64 encode the binary data into something that can be transmitted over text. This works, but using Base64 encoding has several problems though, some of which include:

• Aproximately 33% file size increase. We're taking 3 bytes or binary data and spreading over 4 bytes using only the text writable characters. With large files, this leads to a significant increase.
• It's impossible to do any fancy server side file handling. The entire JSON request will be loaded into the web applications memory. Normally when files are sent using LiquidFiles, the web server (nginx) takes care of the binary file data and the web application only deals with moving files around but the actual file data never passes through the web application. This keeps LiquidFiles fast and efficient. If you where to send a 1Gb file using this method, first it would be 1.3Gb transferred (Base64) and this would be loaded (twice - raw Base64 data and decoded) into the memory of LiquidFiles before it could write it to disk. This is very slow and inefficient.
• With these limitations, the maximum message size you can upload with this method is 100MB. While this should still be plenty for a lot of applications, if you upload files near that size on a regular basis, it's the recommendation to switch to the form based upload mechanism instead.

Request Info and Parameters for JSON based Uploads

To upload files using Base64 Encoding, please see the following example using curl and bash:

cat <<EOF | curl -s -X POST \
     -H "Accept: application/json" \
     --user "nkpIxMK9ucUUE7FvfNpdAf:x" -d @- \
     https://liquidfiles.example.com/shares/project-alpha/folders/root/files

{"file":
  {
    "name":"Presentation1.pptx",
    "data":"`base64 /path/to/Presentation1.pptx`"
  }
}     
EOF