Anaphora v0.8.104 -- anaphora ran without any configuration in place :/

General
1

Hello Paul, There is something wrong with our Anaphora instance, I was off for a while and yesterday morning I see there are no more jobs or any configuration present:

image
image595×373 37 KB

image
image594×207 49.8 KB

I restore a backup from the 14th of March 2025. But I’ve no idea why the configuration was missing :confused:

Any idea ?

kr,
Gautier.

nb: I add the idea of creating an API to export configuration; My idea is to schedule a daily export of the configuration to be able to restore everything quickly and it may help us to deploy anaphora on another instance as well etc.

ps: the instance is now running in the latest version available: v0.8.105

2

Oh no! Can it be that the container was restarted and the config was stored in a file contained in a directory that was not mounted in the host filesystem?

Anyway +1 for the Backup/Restore API.
@gautier.franchini would it be more practical to store that in S3?

3

The docker-compose configuration never change, it is always:

      - ./content/:/usr/src/app/content/:rw # ## Mount PDF reports
      - ./storage/:/usr/src/app/storage/:rw

Of course it could be great to have the possibility to choose the storage type and have an s3 bucket.

BUT in right now the storage type is not a problem; the the problem is that I don’t understand why everything disappeared :confused:

A server reboot, an anaphora container upgrade or docker-compose down/up should have an impact as we store everything on disk.

4

@AntonioV anything comes to mind about this one?

5

Did the system settings also get reset? Like the local users for example.
The system settings are stored in the same folder but in a different file. If they are still the same, this could narrow down the error.

6

I encountered the same issue this morning; I think it came from the ownership:

in my Anaphora working directory: ~/Workspace/ ; I have the following:

content/
storage/
docker-compose.yml
. . .

and the ownership was set to:

  • user: ubuntuadmin
  • group: packer

Whenever anaphora container is top the ownership of storage change from ubuntuadmin:packer to packer:packer ; I think this is the rootcause.

I add the packer group to my ubuntuadmin user groups.

At the end I should probably create a dedicated user to avoid any issue.

kr,
G.

ps: That being said, I install the latest v.0.8.111.

7

That’s an interesting finding — conflicting permissions between the host and container could definitely be the issue.

One possible solution is to use Docker-managed volumes instead of bind mounts. For example:

  - anaphora-content:/usr/src/app/content:rw
  - anaphora-storage:/usr/src/app/storage:rw

This way, Docker handles the ownership and permissions, so you avoid potential conflicts with host file system permissions.

8

Dear All, today I still see that our Anaphora Container is running without any configuration (No more jobs, delivery interface, etc. Nothing!);

  • running version v0.8.117;

I’m not able to restore an anaphora backup file.

Here is my terminal output:

I tried several times, using different backup files, different import options and restart anaphora in a previous version as well : same behavior:

  1. successfully imported message in the GUI
  2. BUT Failed import in the logs
    image
    image1858×279 109 KB

→ resulting in a working anaphora app BUT without any jobs or delivery interfaces and not possibility to restore :confused:

Please advice.

9

Can you try to set this in your nginx config:

With 50M instead.

1 Like
10

Before you do the import of the backup, can you check the storage folder and see if there is an storage.json file? Does it have any content inside?
This could help identify the issue why the data got lost in the first place.

11

Well spotted! It was related to the default value at nginx level! Increasing the limit allowed me to restore a backup file! Thanks @paulb :wink:

12

Hi Paul, yes the storage.json file exists:

image
image1419×510 37.9 KB

Maybe not relevant here but I spot this in the logs as well:

image
image1493×347 47.3 KB

To avoid any side effects, I now run the anaphora container with a dedicated service accompt; Here are my steps if it can help you update the documentation for future users of anaphora :wink:

Create Service Account user (no home dir)

# First, create the group with GID 996
sudo groupadd -g 996 svc_anaphora

# Then, create the user with UID 996, no password, no login shell
sudo useradd -u 996 -g 996 -M -s /bin/false svc_anaphora

# check user
getent passwd svc_anaphora

Explanation:

  • `-u 996``: sets UID to 996
  • `-g 996``: sets primary GID to 996
  • `-M``: do not create a home directory
  • `-s /bin/false``: no login shell (service account)

update docker-compose file

services:
  anaphora-server:
    restart: always
    image: beshultd/anaphora:0.8.117
    ...
    user: "996:996"
    ...

Add svc_anaphora to the docker Group:

sudo usermod -aG docker svc_anaphora
sudo mkdir -p /opt/anaphora
sudo chown -R svc_anaphora:svc_anaphora /opt/anaphora
13

Ok thank you. Can you also check if the system settings are still in place?
Like the LDAP configuration or any local users.

14

It seems they are indeed present:

image
image1425×304 22.2 KB

As a workaround I restored an export from last week, therefore It could be a nice to have in a future release to be able to:

  1. to possibility to select all jobs we want to re-run and have a button to re-run them all one after the other (not in parallel to avoid any issue if targeting the same kibana/elasticsearch instance for example)
  2. An API to schedule a daily export, a daily backup of the anaphora config to be able to restore the service as quickly as possible

Thanks for your help today.

kr,
G.

image
image1872×790 52 KB

15

Thank you for the suggestions. I will add them to our board.
I also have an idea of what could have caused the sudden data loss.
I will start working on a patch for that.

1 Like
16

We identified an issue that could be the cause of the jobs and configuration loss.
In the newest version (0.8.118) is a fix for that.

1 Like
17

Great news! Just to let you know I deployed the latest available (v.0.8.119) and will let you know if the problem occurs again.

image
image1837×1055 71.3 KB

2 Likes
18

So far so good, @gautier.franchini :slight_smile: ?

19

So far so good @sscarduzio :wink:
We’ve been running Anaphora v0.8.120 for about 2 weeks now and everything has been going as expected.

1 Like
20

OK will mark this as resolved now.

1 Like