Crawlab (Master)

Crawlab is a distributed web crawler admin platform supporting multiple programming languages and frameworks. This template deploys the master node with MongoDB.

Configuration

version: "3.8"
services:
  crawlab-master:
    image: crawlabteam/crawlab:latest
    restart: unless-stopped
    environment:
      - CRAWLAB_NODE_MASTER=Y
      - CRAWLAB_MONGO_HOST=mongo
      - CRAWLAB_MONGO_PORT=27017
      - CRAWLAB_MONGO_DB=crawlab
      - CRAWLAB_MONGO_USERNAME=${MONGO_USERNAME}
      - CRAWLAB_MONGO_PASSWORD=${MONGO_PASSWORD}
      - CRAWLAB_MONGO_AUTHSOURCE=admin
    volumes:
      - crawlab-master-data:/root/.crawlab
      - crawlab-app-data:/data
      - crawlab-logs:/var/log/crawlab
    expose:
      - "8080"
      - "9666"
    depends_on:
      - mongo
    deploy:
      resources:
        limits:
          cpus: "${CPU_LIMIT:-0.8}" # 0.8 will limit to 80% of CPU if one core. If 2 cores, you need to put 0.8*2=1.6

  mongo:
    image: mongo:4.2
    restart: unless-stopped
    environment:
      - MONGO_INITDB_ROOT_USERNAME=${MONGO_USERNAME}
      - MONGO_INITDB_ROOT_PASSWORD=${MONGO_PASSWORD}
    volumes:
      - mongo-data:/data/db

volumes:
  crawlab-master-data:
  crawlab-app-data:
  crawlab-logs:
  mongo-data:

[variables]
MONGO_USERNAME = "crawlab"
MONGO_PASSWORD = "${password:24}"
CPU_LIMIT = "0.8"

[config]
env = ["MONGO_USERNAME=${MONGO_USERNAME}", "MONGO_PASSWORD=${MONGO_PASSWORD}", "CPU_LIMIT=${CPU_LIMIT}"]

[[config.domains]]
serviceName = "crawlab-master"
port = 8080
host = "${domain}"

Base64

To import this template in Dokploy: create a Compose service → Advanced → Base64 import and paste the content below:

ewogICJjb21wb3NlIjogInZlcnNpb246IFwiMy44XCJcbnNlcnZpY2VzOlxuICBjcmF3bGFiLW1hc3RlcjpcbiAgICBpbWFnZTogY3Jhd2xhYnRlYW0vY3Jhd2xhYjpsYXRlc3RcbiAgICByZXN0YXJ0OiB1bmxlc3Mtc3RvcHBlZFxuICAgIGVudmlyb25tZW50OlxuICAgICAgLSBDUkFXTEFCX05PREVfTUFTVEVSPVlcbiAgICAgIC0gQ1JBV0xBQl9NT05HT19IT1NUPW1vbmdvXG4gICAgICAtIENSQVdMQUJfTU9OR09fUE9SVD0yNzAxN1xuICAgICAgLSBDUkFXTEFCX01PTkdPX0RCPWNyYXdsYWJcbiAgICAgIC0gQ1JBV0xBQl9NT05HT19VU0VSTkFNRT0ke01PTkdPX1VTRVJOQU1FfVxuICAgICAgLSBDUkFXTEFCX01PTkdPX1BBU1NXT1JEPSR7TU9OR09fUEFTU1dPUkR9XG4gICAgICAtIENSQVdMQUJfTU9OR09fQVVUSFNPVVJDRT1hZG1pblxuICAgIHZvbHVtZXM6XG4gICAgICAtIGNyYXdsYWItbWFzdGVyLWRhdGE6L3Jvb3QvLmNyYXdsYWJcbiAgICAgIC0gY3Jhd2xhYi1hcHAtZGF0YTovZGF0YVxuICAgICAgLSBjcmF3bGFiLWxvZ3M6L3Zhci9sb2cvY3Jhd2xhYlxuICAgIGV4cG9zZTpcbiAgICAgIC0gXCI4MDgwXCJcbiAgICAgIC0gXCI5NjY2XCJcbiAgICBkZXBlbmRzX29uOlxuICAgICAgLSBtb25nb1xuICAgIGRlcGxveTpcbiAgICAgIHJlc291cmNlczpcbiAgICAgICAgbGltaXRzOlxuICAgICAgICAgIGNwdXM6IFwiJHtDUFVfTElNSVQ6LTAuOH1cIiAjIDAuOCB3aWxsIGxpbWl0IHRvIDgwJSBvZiBDUFUgaWYgb25lIGNvcmUuIElmIDIgY29yZXMsIHlvdSBuZWVkIHRvIHB1dCAwLjgqMj0xLjZcblxuICBtb25nbzpcbiAgICBpbWFnZTogbW9uZ286NC4yXG4gICAgcmVzdGFydDogdW5sZXNzLXN0b3BwZWRcbiAgICBlbnZpcm9ubWVudDpcbiAgICAgIC0gTU9OR09fSU5JVERCX1JPT1RfVVNFUk5BTUU9JHtNT05HT19VU0VSTkFNRX1cbiAgICAgIC0gTU9OR09fSU5JVERCX1JPT1RfUEFTU1dPUkQ9JHtNT05HT19QQVNTV09SRH1cbiAgICB2b2x1bWVzOlxuICAgICAgLSBtb25nby1kYXRhOi9kYXRhL2RiXG5cbnZvbHVtZXM6XG4gIGNyYXdsYWItbWFzdGVyLWRhdGE6XG4gIGNyYXdsYWItYXBwLWRhdGE6XG4gIGNyYXdsYWItbG9nczpcbiAgbW9uZ28tZGF0YTpcbiIsCiAgImNvbmZpZyI6ICJbdmFyaWFibGVzXVxuTU9OR09fVVNFUk5BTUUgPSBcImNyYXdsYWJcIlxuTU9OR09fUEFTU1dPUkQgPSBcIiR7cGFzc3dvcmQ6MjR9XCJcbkNQVV9MSU1JVCA9IFwiMC44XCJcblxuW2NvbmZpZ11cbmVudiA9IFtcIk1PTkdPX1VTRVJOQU1FPSR7TU9OR09fVVNFUk5BTUV9XCIsIFwiTU9OR09fUEFTU1dPUkQ9JHtNT05HT19QQVNTV09SRH1cIiwgXCJDUFVfTElNSVQ9JHtDUFVfTElNSVR9XCJdXG5cbltbY29uZmlnLmRvbWFpbnNdXVxuc2VydmljZU5hbWUgPSBcImNyYXdsYWItbWFzdGVyXCJcbnBvcnQgPSA4MDgwXG5ob3N0ID0gXCIke2RvbWFpbn1cIlxuIgp9

Crawlab (Master)

Configuration

Base64

Links

Tags

On this page