TransWikia.com

Celery Multi не получает настройки Django

Stack Overflow на русском Asked on December 21, 2020

Столкнулся со странной проблемой, при запуске пачки работников настройки проекта игнорируются, что приводит к ошибке аутентификации на брокере.

запуск worker

$ celery -A {PROJ} worker -l info
 
 -------------- celery@local v4.4.7 (cliffs)
--- ***** ----- 
-- ******* ---- Linux-5.0.0-32-generic-x86_64-with-LinuxMint-19.3-tricia 2020-08-03 09:27:35
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         {PROJ}:0x7f6f14264410
- ** ---------- .> transport:   amqp://rabbitmq:**@127.0.0.1:5672//
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 16 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> test             exchange=test(direct) key=test
                

[tasks]
  . api.tasks.test_task

[2020-08-03 09:27:35,840: INFO/MainProcess] Connected to amqp://rabbitmq:**@127.0.0.1:5672//
[2020-08-03 09:27:35,845: INFO/MainProcess] mingle: searching for neighbors
[2020-08-03 09:27:36,860: INFO/MainProcess] mingle: all alone
[2020-08-03 09:27:36,875: WARNING/MainProcess] /home/local/.pyenv/{PROJ}/lib/python3.7/site-packages/celery/fixups/django.py:206: UserWarning: Using settings.DEBUG leads to a memory
            leak, never use this setting in production environments!
  leak, never use this setting in production environments!''')
[2020-08-03 09:28:06,022: INFO/MainProcess] Events of group {task} enabled by remote.
^C
worker: Hitting Ctrl+C again will terminate all running tasks!

worker: Warm shutdown (MainProcess)

Как видно на скрине, работник запущен и я могу отследить его состояние в flower и rabbit. Однако при запуске multi, работник падает

$ python -m celery multi start --app={PROJ} --pidfile=/tmp/%n.pid --logfile=/tmp/%n.log 1 -Q default -E

Лог работника

$ cat celery1.log 
[2020-08-03 12:25:53,702: CRITICAL/MainProcess] Unrecoverable error: AccessRefused(403, 'ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.', (0, 0), '')
Traceback (most recent call last):
  File "/home/local/.pyenv/{PROJ}/lib/python3.7/site-packages/celery/worker/worker.py", line 208, in start
    self.blueprint.start(self)
  File "/home/local/.pyenv/{PROJ}/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/home/local/.pyenv/{PROJ}/lib/python3.7/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/home/local/.pyenv/{PROJ}/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/home/local/.pyenv/{PROJ}/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/home/local/.pyenv/{PROJ}/lib/python3.7/site-packages/celery/worker/consumer/connection.py", line 23, in start
    c.connection = c.connect()
  File "/home/local/.pyenv/{PROJ}/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 405, in connect
    conn = self.connection_for_read(heartbeat=self.amqheartbeat)
  File "/home/local/.pyenv/{PROJ}/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 412, in connection_for_read
    self.app.connection_for_read(heartbeat=heartbeat))
  File "/home/local/.pyenv/{PROJ}/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 439, in ensure_connected
    callback=maybe_shutdown,
  File "/home/local/.pyenv/{PROJ}/lib/python3.7/site-packages/kombu/connection.py", line 389, in ensure_connection
    self._ensure_connection(*args, **kwargs)
  File "/home/local/.pyenv/{PROJ}/lib/python3.7/site-packages/kombu/connection.py", line 445, in _ensure_connection
    callback, timeout=timeout
  File "/home/local/.pyenv/{PROJ}/lib/python3.7/site-packages/kombu/utils/functional.py", line 344, in retry_over_time
    return fun(*args, **kwargs)
  File "/home/local/.pyenv/{PROJ}/lib/python3.7/site-packages/kombu/connection.py", line 874, in _connection_factory
    self._connection = self._establish_connection()
  File "/home/local/.pyenv/{PROJ}/lib/python3.7/site-packages/kombu/connection.py", line 809, in _establish_connection
    conn = self.transport.establish_connection()
  File "/home/local/.pyenv/{PROJ}/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
    conn.connect()
  File "/home/local/.pyenv/{PROJ}/lib/python3.7/site-packages/amqp/connection.py", line 320, in connect
    self.drain_events(timeout=self.connect_timeout)
  File "/home/local/.pyenv/{PROJ}/lib/python3.7/site-packages/amqp/connection.py", line 508, in drain_events
    while not self.blocking_read(timeout):
  File "/home/local/.pyenv/{PROJ}/lib/python3.7/site-packages/amqp/connection.py", line 514, in blocking_read
    return self.on_inbound_frame(frame)
  File "/home/local/.pyenv/{PROJ}/lib/python3.7/site-packages/amqp/method_framing.py", line 55, in on_frame
    callback(channel, method_sig, buf, None)
  File "/home/local/.pyenv/{PROJ}/lib/python3.7/site-packages/amqp/connection.py", line 521, in on_inbound_method
    method_sig, payload, content,
  File "/home/local/.pyenv/{PROJ}/lib/python3.7/site-packages/amqp/abstract_channel.py", line 145, in dispatch_method
    listener(*args)
  File "/home/local/.pyenv/{PROJ}/lib/python3.7/site-packages/amqp/connection.py", line 651, in _on_close
    (class_id, method_id), ConnectionError)
amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.

При этом на брокере я вижу лог

rabbit     | AMQPLAIN login refused: user 'guest' - invalid credentials

При запуске celery multi из докера я выясняю следующее

[2020-08-03 09:24:58,741: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Trying again in 8.00 seconds... (4/100)

И вот тут я понять не могу, какого … он пытается использовать свои собственные и прописанные по дефолту данные аутентификации и хост брокера?

в settings указана настройка

CELERY_BROKER_URL = os.environ.get("DJANGO_CELERY_BROKER_URL", "amqp://rabbitmq:[email protected]:5672/")

{PROJ}
├── asgi.py
├── celery.py
├── __init__.py
├── settings
│   ├── flowerconfig.py
│   ├── __init__.py
├── urls.py
└── wsgi.py

Содержимое celery.py

import os

from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', '{PROJ}.settings')
app = Celery('{PROJ}')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
app.conf.beat_schedule = {
    'test-tasks': {
        'task': 'api.tasks.test_task',
        'schedule': 60.0,
        'enabled': True
    }
}

содержимое __init__.py основного пакета прокета

from .celery import app as celery_app

__all__ = ('celery_app',)

настройки для celery из настроек django

from datetime import datetime

from kombu import Queue, Exchange

# DJANGO SETTINGS ...

# CELERY

CELERY_TASK_QUEUES = [
    Queue('test', Exchange('test'), routing_key='test', queue_arguments={'x-queue-mode': 'lazy'}),
]

CELERY_TASK_DEFAULT_QUEUE = 'default'
CELERY_TASK_DEFAULT_EXCHANGE = 'default'
CELERY_TASK_DEFAULT_ROUTING_KEY = 'default'
CELERY_TASK_QUEUE_MAX_PRIORITY = 10

CELERY_TASK_ROUTES = {
    'api.tasks.test_task': {'queue': 'test'}
}

CELERY_BROKER_URL = os.environ.get("DJANGO_CELERY_BROKER_URL", "amqp://rabbitmq:[email protected]:5672/")
CELERY_IGNORE_RESULT = True
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Europe/Moscow'
CELERY_TASK_DEFAULT_RETRY_DELAY = 30
CELERY_ENABLE_UTC = True
CELERY_WORKER_MAX_MEMORY_PER_CHILD = 60_000  # 60 MB
CELERY_TASK_SERIALIZER = 'json'
CELERY_TASK_TIME_LIMIT = 120
CELERY_TASK_MAX_RETRIES = 10
CELERY_ACKS_LATE = True

# CELERYD

CELERYD_PREFETCH_MULTIPLIER = 1
CELERYD_HIJACK_ROOT_LOGGER = False

LOGGING['handlers']['celery_file_handler'] = {
    'level': 'INFO',
    'class': 'logging.handlers.RotatingFileHandler',
    'formatter': 'task_logger',
    'filename': os.path.join(BASE_DIR, 'logs', 'celery', '{}.log'.format(datetime.now().date())),
    'maxBytes': 1024000,
    'backupCount': 3,
}
LOGGING['loggers']['celery.tasks'] = {
    'handlers': ['celery_file_handler', 'console'],
    'level': 'DEBUG',
    'propagate': False
}

Так как settings.py был переделан в пакет, то BASE_DIR была изменена на

BASE_DIR = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

И сервис из докер композ файла

services:
  celery:
    build: .
    container_name: celery
    working_dir: /app
    volumes:
      - ./api:/app/api
      - ./bit:/app/bit
      - ./btcmirror:/app/btcmirror
      - ./lending:/app/lending
      - ./logs:/app/logs
      - ./manage.py:/app/manage.py
    environment:
      DJANGO_DATABASE_HOST: "db"
      DJANGO_DATABASE_PORT: "5432"
      DJANGO_DATABASE_NAME: "default"
      DJANGO_DATABASE_USER: "default"
      DJANGO_DATABASE_PASSWORD: "default"
      # Celery
      DJANGO_CELERY_BROKER_URL: "amqp://rabbitmq:rabbitmq@rabbit:5672"
      # Flower
      DJANGO_FLOWER_BROKER_URL: "http://rabbitmq:rabbitmq@rabbit:15672/api/"
      DJANGO_FLOWER_BASIC_AUTH: "admin:admin"
    deploy:
      resources:
        limits:
          memory: 1G
        reservations:
          memory: 600M
    command: bash -c "
      rm -rf /tmp/celery*.pid &
      python -m celery multi start --app={PROJ} --pidfile=/tmp/%n.pid --logfile=/app/logs/celery/%n.log 1 -Q default -E &&
      python -m celery -A {PROJ} beat --pidfile=/tmp/celerybeat.pid --logfile=/app/logs/celery/beat.log -l debug --scheduler django_celery_beat.schedulers:DatabaseScheduler --detach &&
      python -m celery flower -A {PROJ} --conf=/app/{PROJ}/settings/flowerconfig.py --address=0.0.0.0 --port=5555"
    ports:
      - "127.0.0.1:5555:5555"
    links:
      - rabbit
      - db
    depends_on:
      - rabbit
      - db

Итог, при запуске celery worker, все работает так как описано в документации и я вижу работника в flower и он выполняет работу, а при – celery multi настройки мои игнорируются и я получаю ошибку аутентификации на брокере потому что по умолчанию используются guest:guest@localhost:5672. При этом не важно где это происходит на хосте или на докере. Что я могу сделать чтобы решить эту проблему? или хотя бы в какую сторону копать для того чтобы понять в чем суть этой проблемы?

P.S. Видимо моя проблема должна стать issue для celery. Я использовал пакеты

amqp==2.6.1
billiard==3.6.3.0
celery==4.4.7
flower==0.9.5
kombu==4.6.11
tornado==6.0.4

Поменял версию celery на 4.3.0

celery==4.3.0
amqp==2.6.1
billiard==3.6.3.0
flower==0.9.3
kombu==4.6.11
tornado==5.1.1

И о чудо, все заработало.

celerydjangopythonpython 3.x

Add your own answers!

Ask a Question

Get help from others!

Recent Answers

Recent Questions

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP