Crawlab (Worker)

Crawlab worker node for distributed web crawling. Connects to an existing Crawlab master node via gRPC.

Configuration

version: "3.8"
services:
  crawlab-worker:
    image: crawlabteam/crawlab:latest
    restart: unless-stopped
    environment:
      - CRAWLAB_NODE_MASTER=N
      - CRAWLAB_GRPC_ADDRESS=${MASTER_GRPC_ADDRESS}
      - CRAWLAB_FS_FILER_URL=${MASTER_FILER_URL}
    volumes:
      - crawlab-worker-data:/root/.crawlab
      - crawlab-worker-app:/data
    deploy:
      resources:
        limits:
          cpus: "${CPU_LIMIT:-0.8}" # 0.8 will limit to 80% of CPU if one core. If 2 cores, you need to put 0.8*2=1.6

volumes:
  crawlab-worker-data:
  crawlab-worker-app:

[variables]
MASTER_GRPC_ADDRESS = "<master_node_ip>:9666"
MASTER_FILER_URL = "http://<master_node_ip>:8080/api/filer"
CPU_LIMIT = "0.8"

[config]
env = ["MASTER_GRPC_ADDRESS=${MASTER_GRPC_ADDRESS}", "MASTER_FILER_URL=${MASTER_FILER_URL}", "CPU_LIMIT=${CPU_LIMIT}"]

Base64

To import this template in Dokploy: create a Compose service → Advanced → Base64 import and paste the content below:

ewogICJjb21wb3NlIjogInZlcnNpb246IFwiMy44XCJcbnNlcnZpY2VzOlxuICBjcmF3bGFiLXdvcmtlcjpcbiAgICBpbWFnZTogY3Jhd2xhYnRlYW0vY3Jhd2xhYjpsYXRlc3RcbiAgICByZXN0YXJ0OiB1bmxlc3Mtc3RvcHBlZFxuICAgIGVudmlyb25tZW50OlxuICAgICAgLSBDUkFXTEFCX05PREVfTUFTVEVSPU5cbiAgICAgIC0gQ1JBV0xBQl9HUlBDX0FERFJFU1M9JHtNQVNURVJfR1JQQ19BRERSRVNTfVxuICAgICAgLSBDUkFXTEFCX0ZTX0ZJTEVSX1VSTD0ke01BU1RFUl9GSUxFUl9VUkx9XG4gICAgdm9sdW1lczpcbiAgICAgIC0gY3Jhd2xhYi13b3JrZXItZGF0YTovcm9vdC8uY3Jhd2xhYlxuICAgICAgLSBjcmF3bGFiLXdvcmtlci1hcHA6L2RhdGFcbiAgICBkZXBsb3k6XG4gICAgICByZXNvdXJjZXM6XG4gICAgICAgIGxpbWl0czpcbiAgICAgICAgICBjcHVzOiBcIiR7Q1BVX0xJTUlUOi0wLjh9XCIgIyAwLjggd2lsbCBsaW1pdCB0byA4MCUgb2YgQ1BVIGlmIG9uZSBjb3JlLiBJZiAyIGNvcmVzLCB5b3UgbmVlZCB0byBwdXQgMC44KjI9MS42XG5cbnZvbHVtZXM6XG4gIGNyYXdsYWItd29ya2VyLWRhdGE6XG4gIGNyYXdsYWItd29ya2VyLWFwcDpcbiIsCiAgImNvbmZpZyI6ICJbdmFyaWFibGVzXVxuTUFTVEVSX0dSUENfQUREUkVTUyA9IFwiPG1hc3Rlcl9ub2RlX2lwPjo5NjY2XCJcbk1BU1RFUl9GSUxFUl9VUkwgPSBcImh0dHA6Ly88bWFzdGVyX25vZGVfaXA+OjgwODAvYXBpL2ZpbGVyXCJcbkNQVV9MSU1JVCA9IFwiMC44XCJcblxuW2NvbmZpZ11cbmVudiA9IFtcIk1BU1RFUl9HUlBDX0FERFJFU1M9JHtNQVNURVJfR1JQQ19BRERSRVNTfVwiLCBcIk1BU1RFUl9GSUxFUl9VUkw9JHtNQVNURVJfRklMRVJfVVJMfVwiLCBcIkNQVV9MSU1JVD0ke0NQVV9MSU1JVH1cIl1cbiIKfQ==

Crawlab (Worker)

Configuration

Base64

Links

Tags

On this page