Seaf-server to dtable-storage-server migration errors

mikeboiko · June 19, 2024, 5:15pm

I’m upgrading an old SeaTable server instance and I ran into errors when following the v.3.0 Extra Upgrade Notices.

What I’ve done so far to upgrade from 2.5.0 to 3.0.0:

I switched over to the “new style” of docker-compose deployment by following Switch Installation Method (v4.3) - SeaTable Admin Manual
I successfully upgraded to 3.0.0 by following all the extra upgrade notice steps and running commands such as docker exec -it seatable-server /shared/seatable/scripts/seatable.sh upgrade-sql 3.0

I’m getting stuck on the last step of Extra Upgrade Notices - SeaTable Admin Manual

When I run these commands:

# list number of bases that are not stored in storage-server
docker exec -it seatable-server /templates/migrate_bases.sh --list

# migrate 10 bases to storage-server (repeat this command until all bases are migrated)
docker exec -it seatable-server /templates/migrate_bases.sh --migrate 10

I get:

╭─mike@app ~/temp/seatable
╰─➤  docker exec -it seatable-server /templates/migrate_bases.sh --list         126 ↵
/opt/seatable/seatable-server-latest/dtable-web/thirdpart/selenium/webdriver/firefox/firefox_profile.py:208: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if setting is None or setting is '':
[2024-06-19 16:50:16.034777] Start listing bases in seaf-server.

[2024-06-19 16:50:16.237889] Found 12 bases in seaf-server.

[2024-06-19 16:50:16.240845] List 10 bases info:
Traceback (most recent call last):
  File "/opt/seatable/seatable-server-latest/seafile/lib/python3/site-packages/pysearpc/named_pipe.py", line 85, in _get_transport
    transport = self._pool.get(False)
  File "/usr/lib/python3.8/queue.py", line 167, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/opt/seatable/seatable-server-4.0.0/dtable-web/seahub/dtable/management/commands/migrate_bases_to_storage_server.py", line 48, in handle
    self.list_bases()
  File "/opt/seatable/seatable-server-4.0.0/dtable-web/seahub/dtable/management/commands/migrate_bases_to_storage_server.py", line 94, in list_bases
    file_size = self._get_file_size(dtable) // 1024
  File "/opt/seatable/seatable-server-4.0.0/dtable-web/seahub/dtable/management/commands/migrate_bases_to_storage_server.py", line 103, in _get_file_size
    repo = seafile_api.get_repo(repo_id)
  File "/opt/seatable/seatable-server-latest/seafile/lib/python3/site-packages/seaserv/api.py", line 105, in get_repo
    return seafserv_threaded_rpc.get_repo(repo_id)
  File "/opt/seatable/seatable-server-latest/seafile/lib/python3/site-packages/pysearpc/client.py", line 125, in newfunc
    ret_str = self.call_remote_func_sync(fcall_str)
  File "/opt/seatable/seatable-server-latest/seafile/lib/python3/site-packages/pysearpc/named_pipe.py", line 97, in call_remote_func_sync
    transport = self._get_transport()
  File "/opt/seatable/seatable-server-latest/seafile/lib/python3/site-packages/pysearpc/named_pipe.py", line 87, in _get_transport
    transport = self._create_transport()
  File "/opt/seatable/seatable-server-latest/seafile/lib/python3/site-packages/pysearpc/named_pipe.py", line 80, in _create_transport
    transport.connect()
  File "/opt/seatable/seatable-server-latest/seafile/lib/python3/site-packages/pysearpc/named_pipe.py", line 44, in connect
    self.pipe.connect(self.socket_path)
ConnectionRefusedError: [Errno 111] Connection refused
╭─mike@app ~/temp/seatable
╰─➤                                                                                                                                                                       1 ↵
╭─mike@app ~/temp/seatable
╰─➤  docker exec -it seatable-server /templates/migrate_bases.sh --migrate 10                                                                                           130 ↵
[2024-06-19 16:54:11.210603] Start migrating bases.

[2024-06-19 16:54:11.227861] Load 10 bases in seaf-server.

[2024-06-19 16:54:11.358577] Migrate base Test failed, UUID: d92ab7e5-424c-4735-8bad-3a2a7583214b, error: [Errno 111] Connection refused
[2024-06-19 16:54:11.460468] Migrate base Finances-Family failed, UUID: 7b4f35cb-f543-4780-b94c-6e993f0f9a2c, error: [Errno 111] Connection refused
[2024-06-19 16:54:11.561970] Migrate base Rent failed, UUID: ea96a476-48d8-4841-986f-bb797d8a5319, error: [Errno 111] Connection refused
[2024-06-19 16:54:11.663757] Migrate base Accounting failed, UUID: 5f5042e3-c44a-484b-ab51-5e155169a5a9, error: [Errno 111] Connection refused
[2024-06-19 16:54:11.765720] Migrate base Health failed, UUID: 7443840c-33fe-449d-be4c-28b225b9d607, error: [Errno 111] Connection refused
[2024-06-19 16:54:11.869658] Migrate base Jesse failed, UUID: bbee0cea-7361-47d5-a06c-fea695f733d5, error: [Errno 111] Connection refused
[2024-06-19 16:54:11.971566] Migrate base Inventory failed, UUID: e44cca81-332c-46a2-92c4-8e3efa3d4dc7, error: [Errno 111] Connection refused
[2024-06-19 16:54:12.073514] Migrate base IMB Controls failed, UUID: 11619fe1-b231-4e04-a119-4eaf26fbd00b, error: [Errno 111] Connection refused
[2024-06-19 16:54:12.175278] Migrate base ByteHound failed, UUID: d477920b-e7fd-43ac-a454-58b058d46f33, error: [Errno 111] Connection refused
[2024-06-19 16:54:12.276844] Migrate base Misc failed, UUID: a6eec87f-6a7f-4641-a0a8-4a752aa16b87, error: [Errno 111] Connection refused

[2024-06-19 16:54:12.277043] Finish migrating bases.

I wasn’t planning on using SeaFile with my SeaTable instance. Is this a requirement or can I get around this issue in a different way?

On a side note, when I used the docker image, seatable/seatable-developer:3.0.0, it didn’t contain /templates/migrate_bases.sh, so I had to use seatable/seatable-developer:4.0.0

Thank you very much for your help!
Mike

Edit: not sure why this is being flagged as spam. If anyone has any tips, please let me know.

cdb · June 19, 2024, 8:36pm

This is the relevant error message. Please check all configuration files, if you really updated the host correctly during the switch to the new deployment method. Did you restart SeaTable after that?

mikeboiko · June 19, 2024, 8:58pm

Hi, thank you for your reply.

I updated my host from db to mariadb in all the config files specified in the instructions:

╭─mike@app /opt/seatable-server/seatable/conf
╰─➤  ag mariadb
dtable_server_config.json
2:  "host": "mariadb",

dtable_web_settings.py
5:            'HOST': 'mariadb',

dtable-db.conf
12:host = mariadb

dtable-events.conf
4:host = mariadb

ccnet.conf
7:HOST = mariadb

seafile.conf
7:host = mariadb

I also restarted the server and everything is working properly. The system info page is showing 3.0.0 as the version.

Anything else I could try?

For the record, this is my seatable-server.yml:

---
services:
  seatable-server:
    # image: ${SEATABLE_IMAGE:-seatable/seatable-enterprise:4.4.9}
    image: seatable/seatable-developer:3.0.0
    restart: unless-stopped
    container_name: seatable-server
    volumes:
      - "/opt/seatable-server:/shared"
      # - type: bind
      # source: "./seatable-license.txt"
      # target: "/shared/seatable/seatable-license.txt"
      # read_only: ${SEATABLE_LICENSE_FORCE_READ_ONLY:-false}
    ports:
      - 8081:80
    environment:
      - DB_HOST=mariadb
      - DB_ROOT_PASSWD=${SEATABLE_MYSQL_ROOT_PASSWORD:?Variable is not set or empty}
      - SEATABLE_SERVER_HOSTNAME=${SEATABLE_SERVER_HOSTNAME:?Variable is not set or empty}
      - SEATABLE_SERVER_PROTOCOL=${SEATABLE_SERVER_PROTOCOL:-https}
      - SEATABLE_ADMIN_EMAIL=${SEATABLE_ADMIN_EMAIL:?Variable is not set or empty}
      - SEATABLE_ADMIN_PASSWORD=${SEATABLE_ADMIN_PASSWORD:?Variable is not set or empty}
      - TIME_ZONE=${TIME_ZONE}
      - PYTHON_SCHEDULER_URL=${PYTHON_SCHEDULER_URL:-http://python-scheduler}
      - PYTHON_SCHEDULER_AUTH_TOKEN=${PYTHON_SCHEDULER_AUTH_TOKEN:-}
      - REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
      - SEATABLE_EMAIL_USE_TLS=${SEATABLE_EMAIL_USE_TLS:-}
      - SEATABLE_EMAIL_HOST=${SEATABLE_EMAIL_HOST:-}
      - SEATABLE_EMAIL_HOST_USER=${SEATABLE_EMAIL_HOST_USER:-}
      - SEATABLE_EMAIL_HOST_PASSWORD=${SEATABLE_EMAIL_HOST_PASSWORD:-}
      - SEATABLE_EMAIL_PORT=${SEATABLE_EMAIL_PORT:-}
      - SEATABLE_DEFAULT_FROM_EMAIL=${SEATABLE_DEFAULT_FROM_EMAIL:-}
      - SEATABLE_SERVER_EMAIL=${SEATABLE_SERVER_EMAIL:-}
      - SEATABLE_SHOW_TEMPLATES_LINK=${SEATABLE_SHOW_TEMPLATES_LINK:-}
      - SEATABLE_TEMPLATE_BASE_API_TOKEN=${SEATABLE_TEMPLATE_BASE_API_TOKEN:-}
      - SEATABLE_TEMPLATE_TABLE_NAME=${SEATABLE_TEMPLATE_TABLE_NAME:-}
      - SEATABLE_ENABLE_CREATE_BASE_FROM_TEMPLATE=${SEATABLE_ENABLE_CREATE_BASE_FROM_TEMPLATE:-}
      - SEATABLE_HELP_LINK=${SEATABLE_HELP_LINK:-https://docs.seatable.io}
      - SEATABLE_LOG_LEVEL=${SEATABLE_LOG_LEVEL:-INFO}
    labels:
      caddy: ${SEATABLE_SERVER_PROTOCOL:-https}://${SEATABLE_SERVER_HOSTNAME:?Variable is not set or empty}
      caddy.reverse_proxy: "{{upstreams 80}}"
      # Default security headers
      caddy.header.Strict-Transport-Security: "`max-age=31536000;`"
      caddy.header.Referrer-Policy: "same-origin"
      caddy.header.X-XSS-Protection: "`1; mode=block`"
      caddy.header.X-Content-Type-Options: "nosniff"
      caddy.header.X-Frame-Options: "SAMEORIGIN"
      caddy.header.Content-Security-Policy: "`
        block-all-mixed-content;
        default-src 'none';
        style-src 'unsafe-inline' 'self';
        script-src 'unsafe-inline' 'unsafe-eval' 'self';
        script-src-elem 'unsafe-inline' 'self' ${SEATABLE_SERVER_HOSTNAME}:${ONLYOFFICE_PORT:-6233} maps.googleapis.com;
        font-src 'self' data:;
        img-src 'self' data: blob: https: market.seatable.io mt0.google.com maps.googleapis.com maps.gstatic.com;
        form-action 'self' ${SEATABLE_SERVER_HOSTNAME}:${COLLABORA_PORT:-6232};
        connect-src 'self' market.seatable.io https:;
        frame-src 'self' ${SEATABLE_SERVER_HOSTNAME}:${COLLABORA_PORT:-6232} ${SEATABLE_SERVER_HOSTNAME}:${ONLYOFFICE_PORT:-6233};
        frame-ancestors 'self';
        worker-src 'self' blob:;
        manifest-src 'self';
        object-src 'self';
        base-uri 'self'
        `"
      # Allow iframes for some directories
      caddy.route_0: /dtable/view-external-links/*
      caddy.route_0.header.X-Frame-Options: "ALLOWALL"
      caddy.route_0.header.-Content-Security-Policy: ""
      caddy.route_1: /dtable/external-links/*
      caddy.route_1.header.X-Frame-Options: "ALLOWALL"
      caddy.route_1.header.-Content-Security-Policy: ""
      caddy.route_2: /dtable/external-apps/*
      caddy.route_2.header.X-Frame-Options: "ALLOWALL"
      caddy.route_2.header.-Content-Security-Policy: ""
      caddy.route_3: /dtable/forms/*
      caddy.route_3.header.X-Frame-Options: "ALLOWALL"
      caddy.route_3.header.-Content-Security-Policy: ""

    depends_on:
      mariadb:
        condition: service_healthy
      memcached:
        condition: service_healthy
      redis:
        condition: service_healthy
    networks:
      - frontend-net
      - backend-seatable-net
    # healthcheck specifically for dtable-web
    healthcheck:
      test: ["CMD-SHELL", "curl --fail http://localhost:8000 || exit 1"]
      interval: 20s
      retries: 3
      start_period: 30s
      timeout: 10s

  mariadb:
    image: ${SEATABLE_DB_IMAGE:-mariadb:10.11.7-jammy}
    restart: unless-stopped
    container_name: mariadb
    environment:
      - MYSQL_ROOT_PASSWORD=${SEATABLE_MYSQL_ROOT_PASSWORD:?Variable is not set or empty}
      - MYSQL_LOG_CONSOLE=true
      - MARIADB_AUTO_UPGRADE=1
    volumes:
      - "/opt/mariadb:/var/lib/mysql"
    networks:
      - backend-seatable-net
    healthcheck:
      test:
        [
          "CMD",
          "/usr/local/bin/healthcheck.sh",
          "--connect",
          "--mariadbupgrade",
          "--innodb_initialized",
        ]
      interval: 20s
      retries: 3
      start_period: 30s
      timeout: 10s
      # On older database containers without healthcheck users present you might need to create them manually,
      # otherwise the container stays unhealthy.
      # more info at https://admin.seatable.io/upgrade/extra-upgrade-notice/

  memcached:
    image: ${SEATABLE_MEMCACHED_IMAGE:-memcached:1.6.27-bookworm}
    restart: unless-stopped
    container_name: memcached
    entrypoint: memcached -m 256
    networks:
      - backend-seatable-net
    healthcheck:
      test: ["CMD-SHELL", "timeout 2 bash -c '</dev/tcp/localhost/11211'"]
      interval: 20s
      retries: 3
      timeout: 5s

  redis:
    image: ${SEATABLE_REDIS_IMAGE:-redis:7.2.4-bookworm}
    restart: unless-stopped
    container_name: redis
    networks:
      - backend-seatable-net
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 20s
      retries: 3
      timeout: 5s

networks:
  frontend-net:
    name: frontend-net
  backend-seatable-net:
    name: backend-seatable-net

cdb · June 20, 2024, 7:58am

Is there a reason, that you are using 3.0.0? This version is two years old.

mikeboiko · June 20, 2024, 2:15pm

I haven’t updated for a long time, so I’m running 2.5.
I was trying to incrementally update to 4.4, and got stuck on the 3.0 migrations.
That said, I spent way too many hours trying different things yesterday and it seems that something got messed up with my database.
I’m going to abandon this plan and will just deploy a new instance of SeaTable, using the latest version. I will then copy my bases over by hand. Unfortunately, the .dtable export files between versions 4.4 and 2.5 are incompatible, so I’ll have to do it manually.

Anyway, feel free to close this thread. Thanks again for your help.