Nextcloud NixOS Container

Updated: at 11:57 PM

Overview

The following goes over nextcloud setup in declarative nixos-container with PSQL. I’ve set requirements which includes:

Outline:

Dev Log

There were several learning experiences for myself, but I’ll focus on the more tricky ones with fewer solutions online.

Debugging each issue was a slow process, especially with changes to the container config as I’m doing a nixos-rebuild switch. It can take up to 2 minutes to rebuild my system with changes.

Networking and containers.nextcloud

The wiki page for nixos-containers uses nextcloud as an example, yet it’s ambiguous, misleading and incomplete in describing setting up the containers own network stack and exposing bound $ip:$port to host. This guy actually gets us going in the right direction https://blog.beardhatcode.be/2020/12/Declarative-Nixos-Containers.html.

We define our nextcloud container as

containers.nextcloud = {
  autoStart = true;
  privateNetwork = true;
  hostAddress = "192.168.100.2";
  localAddress = "192.168.100.11";
};

Find our host networking interface with ifconfig. Oftentimes is eth0, or enp something, then set

networking.nat = {
  enable = true;
  internalInterfaces = ["ve-+"];
  externalInterface = "enp42s0";
};

nextcloud will bind to all within the container on port 80 for ipv4 and ipv6 from the looks of it

[root@nextcloud:~]# cat /etc/hosts
127.0.0.1 localhost
::1 localhost
127.0.0.2 nextcloud
::1 nextcloud

[root@nextcloud:~]# netstat -tuln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN     
tcp6       0      0 :::80                   :::*                    LISTEN     
tcp6       0      0 ::1:5432                :::*                    LISTEN

We can infer process binding to the ips on port 5432 is psql since 5432 is its default port.

There’s several small configs we should be setting within the container config that includes:

Open port 80 within the container (tcp and tcp6)

networking = {
  firewall = {
    enable = true;
    allowedTCPPorts = [ 80 ];
  };
};

Though we’ll only be using ipv4 ip from the container.

Admin User

I use git-crypt in my nixcfg/ (system flake.nix) for secret file management. So within the container, I link to it in the nix store with

systemd.tmpfiles.rules = [
  "L+ /var/lib/nextcloud/admin-pw          0440 nextcloud nextcloud - ${adminPwFile}"
];

where adminPwFile is declared derivation to a direct file within my flake.nix directory in a git-crypt encrypted directory commited in git. It’s generally recommended to use sops-nix or agenix, and to not store plaintext secrets in the nix store. My management of secrets here is simplified as sops-nix won’t let you commit multiple secret files. Leveraging 1 file decrypted at development and build time is very conveinent, e.g.:

let
  adminPwFile = ../.secrets/elk/nextcloud-admin-pw;
in {}

By now, we should be able to access http://192.168.100.11 from the host. We can verify the ip of the nextcloud container with nixos-container show-ip nextcloud

PSQL

Not using the default sqlite is encouraged by nextcloud as per their warning when visiting the default landing page, /login

Performance warning You chose SQLite as database. SQLite should only be used for minimal and development instances. For production we recommend a different database backend. If you use clients for file syncing, the use of SQLite is highly discouraged.

If you’re having troubles migrating to PSQL, reference my config below and visit the Debugging section.

We:

Things are setup, but of we need to test and verify we have a plan and the backup + restore process works :)

Bind Mount between Host and Container

I bind the data dir in container, /var/lib/nextcloud/data to host /mnt/nextcloud-data with read + write permissions.

I use systemd.tmpfiles.rules to create the directory on the host with the appropriate file privileges and user:group.

One improvement I made to my system flake.nix along the way is explicitly setting my uid and gid of my personal user to 1000:100 for consistency. I had some weirdness going on with nixos where using

systemd.tmpfiles.rules = [
  "d /mnt/nextcloud-data 0755 j users"
];

would set 999:999 but then my uid and gid would change and some other : would have 999:999 resulting in me not being about to access my own /mnt/nextcloud-data without sudo. I then moved my uid and gid to variables passed along to all modules for access as I do with user and primaryGroup to then be able to do this more declaratively and variabely

systemd.tmpfiles.rules = [
  "d /mnt/nextcloud-data 0755 ${toString vars.uid} ${toString vars.gid}"
];

defining vars as a specialArgs at my top-level nixosConfigurations.<host>

Debugging

There were quite a few errors I was running into. nextcloud and the nixos config for it isn’t the best experience especially when switching things around.

As long as we nixos rebuild switch successfully from the host system, we can

sudo nixos-container root-login nextcloud

to then further inspect configs, services, etc. Most of my debugging takes place within the container.

Early on testing connectivity from host, I noticed within the container, I could get a response from

curl http://nextcloud

but not localhost (reference my previous /etc/hosts within container). I would later set my container local ip to the hostName value to supplement this.

Our config dir looks like this

[root@nextcloud:~]# l /var/lib/nextcloud/config/
total 16K
drwxr-x--- 2 nextcloud nextcloud 4.0K Sep  7 14:40 .
drwxr-x--- 5 nextcloud nextcloud 4.0K Sep  7 13:53 ..
-r-------- 1 nextcloud nextcloud    0 Sep  7 14:40 CAN_INSTALL
-rw-r----- 1 nextcloud nextcloud 1.3K Sep  7 14:27 config.php
lrwxrwxrwx 1 root      root        64 Sep  7 13:53 override.config.php -> /nix/store/aa5v311q7c54qibr0nsfzm4xwvr4v237-nextcloud-config.php

The single most common error with setting up nextcloud will be thrown in

[root@nextcloud:~]# systemctl status nextcloud-setup.service 
× nextcloud-setup.service
     Loaded: loaded (/etc/systemd/system/nextcloud-setup.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Sat 2024-09-07 14:10:08 PDT; 10min ago
    Process: 285 ExecStart=/nix/store/4iyi845z4n2rl3frrih8a0f0g16cjn0h-unit-script-nextcloud-setup-start/bin/nextcloud-setup-start (code=exit>
   Main PID: 285 (code=exited, status=1/FAILURE)
        CPU: 106ms

Sep 07 14:10:08 nextcloud systemd[1]: Starting nextcloud-setup.service...
Sep 07 14:10:08 nextcloud nextcloud-setup-start[293]: Nextcloud is not installed - only a limited number of commands are available
Sep 07 14:10:08 nextcloud nextcloud-setup-start[293]:                                      
Sep 07 14:10:08 nextcloud nextcloud-setup-start[293]:   Command "upgrade" is not defined.
Sep 07 14:10:08 nextcloud nextcloud-setup-start[293]:                                      
Sep 07 14:10:08 nextcloud systemd[1]: nextcloud-setup.service: Main process exited, code=exited, status=1/FAILURE
Sep 07 14:10:08 nextcloud systemd[1]: nextcloud-setup.service: Failed with result 'exit-code'.
Sep 07 14:10:08 nextcloud systemd[1]: Failed to start nextcloud-setup.service.

I found many threads in repos issues and forums on the matters, but none go into detail or provided solutions that worked for me at the time of writing, so I write an entire post about it. Unfortunately, we’re facing the anti-pattern of 1 vague error describing a wide variety of issues as seen throughout nextcloud issue threads.

One script we reference is that of the nextcloud-setup.service ExecStart script, i.e.:

[root@nextcloud:~]# cat /nix/store/4iyi845z4n2rl3frrih8a0f0g16cjn0h-unit-script-nextcloud-setup-start/bin/nextcloud-setup-start
<output omitted>

in here, we can see the validation conditions, setup process, and determine what to run to debug points of failures

At times, my /var/lib/nextcloud/config/config.php was empty (0 bytes). I would just delete it then try restarting the nextcloud-setup.service or running the command directly myself, that is,

mv /var/lib/nextcloud/config/config.php temp
sudo -u nextcloud DBPASS="" ADMINPASS="$(<"/var/lib/nextcloud/admin-pw")" \
  /nix/store/yfrk6xybwmaxxdblks0pj4a473j0z819-nextcloud-occ/bin/nextcloud-occ maintenance:install \
    --admin-pass "$ADMINPASS" \
    --admin-user "admin" \
    --data-dir "/var/lib/nextcloud/data" \
    --database "pgsql" \
    --database-host "/run/postgresql" \
    --database-name "nextcloud" \
    --database-pass "$DBPASS" \
    --database-user "nextcloud"

it’s possible that you get this error from the nextcloud-occ command,

The Login is already being used

in which case, you may be facing an issue with the state of nextcloud DB used, in my case, psql.

If you migrated from sqlite (default DB) to psql (and did your data backups), you may need to

sudo -u postgres psql -c "DROP DATABASE nextcloud;"
sudo -u postgres psql -c "CREATE DATABASE nextcloud OWNER nextcloud;"

then try restarting the service or running the nextcloud-occ command above again.

Along the way, we may the error from the nextcloud webpage

Error It looks like you are trying to reinstall your Nextcloud. However the file CAN_INSTALL is missing from your config directory. Please create the file CAN_INSTALL in your config folder to continue.

Within the container we then additionally set a 2nd rule,

systemd.tmpfiles.rules = [
  "L+ /var/lib/nextcloud/admin-pw          0440 nextcloud nextcloud - ${adminPwFile}"
  "f /var/lib/nextcloud/config/CAN_INSTALL 0400 nextcloud nextcloud -"
];

Alternatively, you could try to touch the file then restart the service as needed. The implications of CAN_INSTALL and its impact is up to you to research as I’m just hacking my way through for my own initial requirements without risking any real data. I’ll later remove the 2nd rule for production release.

Production Deployment

My requirements are simple. I’m the only one using this. I’m putting this on a public facing endpoint, but have multiple layers of security, that is:

In the future, I could utilize nextcloud’s server-side or end-to-end encryption, encrypt the directory or mount an encrypted drive for binding host /mnt/nextcloud-data to container nextcloud data. I could consider rolling my own certificate authority for the TLS.

Allowlist my public IP for controlling access with

extraConfig = ''
  allow ${permittedIp};
  deny all;
'';

I add an A record in my zone file for my subdomain, mysubdomain.example.com, to submit to my domain defined in AWS Route 53. Enable TLS for my subdomain with

virtualHosts."mysubdomain.example.com" = {
  enableACME = true; # automated cert management env
  forceSSL = true; # use https, not http
  # ...
};

There’s several other configs for setting up TLS, Let’s Encrypt, and nginx for my domain, but I won’t go into detail here.

Reverse proxy requests to mysubdomain.example.com to 192.168.100.11 on the host with

locations."/" = { # reverse proxy requests to this location on the host
  proxyPass = "http://192.168.100.11:80";
  # ...
};

One improvement I could make abstracting this magic value, 192.168.100.11, to a variable.

But wait, there’s more, errors.

Performance warning You chose SQLite as database. SQLite should only be used for minimal and development instances. For production we recommend a different database backend. If you use clients for file syncing, the use of SQLite is highly discouraged.

I’d infer this implies psql is misconfigured.

● nextcloud-cron.service                   loaded failed failed    nextcloud-cron.service
● nextcloud-setup.service                  loaded failed failed    nextcloud-setup.service
● nextcloud-update-db.service              loaded failed failed    nextcloud-update-db.service

3 services failed while all else is fine

Nextcloud is not installed - only a limited number of commands are available

Fixed with

rm /var/lib/nextcloud/config/config.php && systemctl restart nextcloud-setup.service

Well, nextcloud-setup.service is still failing with the same error, but I’ll check on that later as something additional broke.

I can curl http://nextcloud in container, but I can’t curl http://192.168.100.11 on host. Thinking back to it, I forgot to add crucial config for networking on my elk host to enable the network addressing translation config. That is, I needed to add

networking.interfaces.eht0.name = "ens18";

Unfortunately I don’t have a good solution for making this declarative.

I’m still facing the same problem, let’s see what does work.

[j@elk:~]$ nix-shell -p traceroute
[nix-shell:~]$ sudo traceroute -T -p 80 192.168.100.11
traceroute to 192.168.100.11 (192.168.100.11), 30 hops max, 60 byte packets
 1  nextcloud.containers (192.168.100.11)  0.358 ms  0.057 ms  0.046 ms

TCP works, but not HTTP.

Double checking HTTP response within the container, with curl --verbose http://nextcloud, I find it’s actually returning a 400 and a html response body that has the error message

Access through untrusted domain

Please contact your administrator. If you are an administrator, edit the “trusted_domains” setting in config/config.php like the example in config.sample.php.

I add my subdomain serving nextcloud and the IP assigned to the host for the primary network interface to the trusted_domains list.

services.nextcloud.settings = {
  trusted_domains = [ "192.168.100.11" "mysubdomain.example.com" my-ip ];
};

I forgot a semicolon in my nginx config as suggested in these logs and unreachable endpoint.

[j@elk:~]$ tail -1 /var/log/nginx/error.log
2024/09/08 22:28:44 [emerg] 286614#286614: invalid number of arguments in "allow" directive in /nix/store/mhj7zpk5p8mflyhcmfhyxxx0ky1w815n-nginx.conf:136

That wraps up deploying nextcloud to production for myself.

Future Work

This was more effort with less support than I would have hoped.