How to change the size limit of upload files？

Projects Invenio RDM
1

I have a problem is the uploading will be broken as content too large, even the size of a file is not reach the limits. But this situation only appear in containers mode. When I use local mode, it disappears.

For example, I have change the docker/nginx/conf.d/default.conf as below:

location / {
    uwsgi_pass ui_server;
    include uwsgi_params;
    [......]
    # Max upload size (except for files) is set to 100mb as default.
    client_max_body_size 100m;
  }
  ## Most API
  location /api {
    uwsgi_pass api_server;
    include uwsgi_params;
    [......]
    # Max upload size (except for files) is set to 100mb as default.
    client_max_body_size 100m;
  }
  location ~ /api/records/.+/draft/files/.+/content {
    gzip off;
    uwsgi_pass api_server;
    include uwsgi_params;
    [......]
    # Max upload size for files is set to 50GB (configure as needed).
    client_max_body_size 50G;
  }

And add the lines to invenio.cfg:

APP_RDM_DEPOSIT_FORM_QUOTA = {
    "maxFiles": 100,
    "maxStorage": 40000000000,
}

When I run server with invenio-cli run, I can upload a 10GB filel; But if I run server with invenio-cli containers start --no-demo-data, the progress will break at 2% and F12 check the errer,
image

2

Before digging in nginx configurations, did you check if you have mounted a volume, that can contain large files? It could be simply that you don’t have enough space in the Docker container.
Here it does not look like that we mount a volume.

3

I checked the filesystem. Because I didn’t use s3 file system, so the local storage was used. The path of data is /opt/invenio/var/instance/data/, and it has all the files attaching in old records actually.

image
image733×94 2.72 KB

The free disk space is enough for a 1.89G file, the result is shown:
image
Do you have any idea about the error of “Request Entity Too Large”

4

It could be related to the rate limiter, see discussion in Discord: Discord

Or to a specific nginx config, see this message: Discord

we have three locations......with /api/files having a 50G upload limit, but /api/files is wrong, it should be /api/records/:id/draft/files/content

Can you check your nginx config if it is the same as here?

5

The Discard may not be open due to regional restrictions. The nginx config was the same as github, the only change is adding the 80 port in HTTPS server and discarding it from HTTP server. So my default.conf is look like:

upstream ui_server {
  server web-ui:5000 fail_timeout=0;
}
upstream api_server {
  server web-api:5000 fail_timeout=0;
}

# HTTPS server
server {
  listen 80;
  listen [::]:80;
  listen 443 ssl; # IPv4
  listen [::]:443 ssl; # IPv6
  server_name _;
  charset utf-8;
  keepalive_timeout 5;

  [......]
# Max upload size for files is set to 50GB (configure as needed).
    client_max_body_size 50G;
  }
  # Static content is served directly by nginx and not the application server.
  location /static {
    alias /opt/invenio/var/instance/static;
    autoindex off;
  }
}
6

It really looks like nginx blocking it, given also the URL in your console error… are you 100% positive that this is the config used by nginx?

7

Thank you for the reminder, I check the config again. The config of nginx in InvenioRDM is used like mentioned above, but there is another nginx in the top of the server. The engineer bind our site to InvenioRDM server, so the content size was limited by the site config. After changing the config of site, the large file was uploaded successfully.

1 Like