晓风残月

The incident began with the accidental discovery that the compiled binary of bili-sync was excessively large, reaching an astonishing 26M. Since I had recently used Rust Embed (hereinafter referred to as embed) to package the static files generated by the frontend into the binary, I naturally suspected that the size of the frontend files was the cause. After a brief investigation, I found that the frontend I wrote occupied less than 1M, while the embedded Swagger-UI took up over 10M. However, regardless of the proportions, both were packaged into the binary program via embed, so I needed to explore the compression options for embed.

Compression Provided by Embed#

Looking at the official README of embed, I found that it offers a compression feature flag:

• compression: Compress each file when embedding into the binary. Compression is done via include-flate.

So, does this solve the problem? Not quite. Upon reviewing the relevant documentation, the compression in embed is implemented through include-flate, so let's take a look at the introduction to include-flate:

A variant of include_bytes!/include_str! with compile-time deflation and runtime lazy inflation.
A variant of include_bytes! / include_str! with compile-time compression and runtime lazy decompression.

We know that embedding solutions like embed are essentially wrappers around include_bytes! / include_str!. These macros, when expanded at compile time, convert file data into static data structures included in the binary file. Include-flate is a variant of these two, and we can infer its operational principle from the introduction:

1. Compress the file content at compile time, using include_bytes! to store the compressed data;
2. At runtime, if the file is already decompressed, it returns directly; otherwise, it retrieves the compressed data, decompresses it, caches it, and returns.

This solution is essentially "transparent compression," providing an API identical to include_bytes!, allowing users to be unaware of the internal decompression process. However, the cost is memory waste.

At runtime, the program needs to store both the compressed data in the binary and the original data that is decompressed upon first access. Comparing to an uncompressed solution:

1. Program size: Original data → Compressed data (decreased)
2. Memory usage: Original data → Original data + Compressed data (increased)

While I wanted to reduce the program size, I did not want to waste more memory. To achieve both a smaller program size and reduced memory usage, I needed to find a new solution.

Content-Encoding#

As we know, HTTP itself supports compressing the response body and provides a Content-Encoding header to indicate the compression algorithm used. Considering that the scenario for using embed is hosting static files, we can pre-compress the static files, and the program will only package the compressed files. When the browser requests them, it directly returns the compressed files and sets the appropriate Content-Encoding header, allowing the browser to handle decompression itself. This way, we can achieve both goals: reducing the program size and minimizing memory usage.

Initially, I thought about forking embed to make modifications, but a bunch of exposed cfg! checks and macro concatenations made me reconsider:

Fortunately, after searching, I found a project similar to my needs: rust-embed-for-web. This project pre-packages files in gzip and br compression formats based on embed, with the original file, gzip compressed, and br compressed accessible via .data(), .data_gzip(), and .data_br(), respectively, where .data() is guaranteed to exist, and the other two return Option which can be controlled via macro parameters. It is evident that this project will also occupy additional space, but it optimizes static file transmission through a precompiled method. To achieve space savings, it is sufficient to change the return value of .data() to Option and modify the procedural macro to support the configuration of whether to store the source file.

The code of this project is much clearer than embed, making modifications very convenient. During the modification process, I also used enum_dispatch to replace dynamic dispatch with static dispatch, which should bring some performance improvements; specific changes can be viewed in this commit.

A side note: I found that the implementation of path => file mapping in this project expands the macro to:

match path {
    path1 => file1,
    path2 => file2,
    ...
}

Whereas the original embed expands to:

const ENTRIES: &'static [(&'static str, EmbeddedFile)] = [(path1, file1), (path2, file2), ...];
let position = ENTRIES.binary_search_by_key(&path.as_str(), |entry| entry.0);
position.ok().map(|idx| ENTRIES[idx].1)

I am not very familiar with the specific matching mechanism of the match statement and am unsure which method performs better; if any experts read this, please feel free to provide insights in the comments.

An Exception#

Observant friends should notice that my modifications above not only support the "whether to store the source file" switch but also add an "except" condition. This is because Swagger-UI has a special file that needs to be dynamically replaced by the server:

pub fn serve<'a>(
    path: &str,
    config: Arc<Config<'a>>,
) -> Result<Option<SwaggerFile<'a>>, Box<dyn Error>> {
    let mut file_path = path;

    if file_path.is_empty() || file_path == "/" {
        file_path = "index.html";
    }

    if let Some(file) = SwaggerUiDist::get(file_path) {
        let mut bytes = file.data;
        // The culprit "swagger-initializer.js", how annoying!
        if file_path == "swagger-initializer.js" {
            let mut file = match String::from_utf8(bytes.to_vec()) {
                Ok(file) => file,
                Err(error) => return Err(Box::new(error)),
            };

            file = format_config(config.as_ref(), file)?;

            if let Some(oauth) = &config.oauth {
                match oauth::format_swagger_config(oauth, file) {
                    Ok(oauth_file) => file = oauth_file,
                    Err(error) => return Err(Box::new(error)),
                }
            }

            bytes = Cow::Owned(file.as_bytes().to_vec())
        };

        Ok(Some(SwaggerFile {
            bytes,
            content_type: mime_guess::from_path(file_path)
                .first_or_octet_stream()
                .to_string(),
        }))
    } else {
        Ok(None)
    }
}

Since swagger-initializer.js needs the server to dynamically replace the template content, the original file must be retained. To accommodate this special file, I had to add the preserve_source_except handling.

Merging Back to bili-sync#

After completing the above work, I could merge the changes back into bili-sync. First, modify the parameters of Asset:

#[derive(RustEmbed)]
#[preserve_source = false] // Do not retain the original file
#[gzip = false] // Do not enable gzip (only enable br)
#[folder = "../../web/build"]
struct Asset;

Then, add the Content-Encoding header in the static resource access part:

let Some(content) = Asset::get(path) else {
    return (StatusCode::NOT_FOUND, "404 Not Found").into_response();
};
Response::builder()
    .status(StatusCode::OK)
    .header(
        header::CONTENT_TYPE,
        content.mime_type().as_deref().unwrap_or("application/octet-stream"),
    )
    .header(header::CONTENT_ENCODING, "br")
    // safety: `RustEmbed` will always generate br-compressed files if the feature is enabled
    .body(Body::from(content.data_br().unwrap()))
    .unwrap_or_else(|_| {
        return (StatusCode::INTERNAL_SERVER_ERROR, "500 Internal Server Error").into_response();
    })

After multiple compilation tests, the results are as follows:

The effect is immediate, and the compression results are significant.