A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 01:45 UTC
[MISC] Rework logger to enable pythonic custom logging configuration to be provided (#4273)
Danny Guinther committed
May 1, 2024 at 20:34 UTC
b8afa8b95a4eee008a9b72440620113e5bfbe962
5 files changed
+451
-48
examples/logging_configuration.md
new
+178
@@ -0,0 +1,178 @@
1
+# Logging Configuration
2
+
3
+vLLM leverages Python's `logging.config.dictConfig` functionality to enable
4
+robust and flexible configuration of the various loggers used by vLLM.
5
+
6
+vLLM offers two environment variables that can be used to accommodate a range
7
+of logging configurations that range from simple-and-inflexible to
8
+more-complex-and-more-flexible.
9
+
10
+- No vLLM logging (simple and inflexible)
11
+ - Set `VLLM_CONFIGURE_LOGGING=0` (leaving `VLLM_LOGGING_CONFIG_PATH` unset)
12
+- vLLM's default logging configuration (simple and inflexible)
13
+ - Leave `VLLM_CONFIGURE_LOGGING` unset or set `VLLM_CONFIGURE_LOGGING=1`
14
+- Fine-grained custom logging configuration (more complex, more flexible)
15
+ - Leave `VLLM_CONFIGURE_LOGGING` unset or set `VLLM_CONFIGURE_LOGGING=1` and
16
+ set `VLLM_LOGGING_CONFIG_PATH=<path-to-logging-config.json>`
17
+
18
+
19
+## Logging Configuration Environment Variables
20
+
21
+### `VLLM_CONFIGURE_LOGGING`
22
+
23
+`VLLM_CONFIGURE_LOGGING` controls whether or not vLLM takes any action to
24
+configure the loggers used by vLLM. This functionality is enabled by default,
25
+but can be disabled by setting `VLLM_CONFIGURE_LOGGING=0` when running vLLM.
26
+
27
+If `VLLM_CONFIGURE_LOGGING` is enabled and no value is given for
28
+`VLLM_LOGGING_CONFIG_PATH`, vLLM will use built-in default configuration to
29
+configure the root vLLM logger. By default, no other vLLM loggers are
30
+configured and, as such, all vLLM loggers defer to the root vLLM logger to make
31
+all logging decisions.
32
+
33
+If `VLLM_CONFIGURE_LOGGING` is disabled and a value is given for
34
+`VLLM_LOGGING_CONFIG_PATH`, an error will occur while starting vLLM.
35
+
36
+### `VLLM_LOGGING_CONFIG_PATH`
37
+
38
+`VLLM_LOGGING_CONFIG_PATH` allows users to specify a path to a JSON file of
39
+alternative, custom logging configuration that will be used instead of vLLM's
40
+built-in default logging configuration. The logging configuration should be
41
+provided in JSON format following the schema specified by Python's [logging
42
+configuration dictionary
43
+schema](https://docs.python.org/3/library/logging.config.html#dictionary-schema-details).
44
+
45
+If `VLLM_LOGGING_CONFIG_PATH` is specified, but `VLLM_CONFIGURE_LOGGING` is
46
+disabled, an error will occur while starting vLLM.
47
+
48
+
49
+## Examples
50
+
51
+### Example 1: Customize vLLM root logger
52
+
53
+For this example, we will customize the vLLM root logger to use
54
+[`python-json-logger`](https://github.com/madzak/python-json-logger) to log to
55
+STDOUT of the console in JSON format with a log level of `INFO`.
56
+
57
+To begin, first, create an appropriate JSON logging configuration file:
58
+
59
+**/path/to/logging_config.json:**
60
+
61
+```json
62
+{
63
+ "formatters": {
64
+ "json": {
65
+ "class": "pythonjsonlogger.jsonlogger.JsonFormatter"
66
+ }
67
+ },
68
+ "handlers": {
69
+ "console": {
70
+ "class" : "logging.StreamHandler",
71
+ "formatter": "json",
72
+ "level": "INFO",
73
+ "stream": "ext://sys.stdout"
74
+ }
75
+ },
76
+ "loggers": {
77
+ "vllm": {
78
+ "handlers": ["console"],
79
+ "level": "INFO",
80
+ "propagate": false
81
+ }
82
+ },
83
+ "version": 1
84
+}
85
+```
86
+
87
+Next, install the `python-json-logger` package if it's not already installed:
88
+
89
+```bash
90
+pip install python-json-logger
91
+```
92
+
93
+Finally, run vLLM with the `VLLM_LOGGING_CONFIG_PATH` environment variable set
94
+to the path of the custom logging configuration JSON file:
95
+
96
+```bash
97
+VLLM_LOGGING_CONFIG_PATH=/path/to/logging_config.json \
98
+ python3 -m vllm.entrypoints.openai.api_server \
99
+ --max-model-len 2048 \
100
+ --model mistralai/Mistral-7B-v0.1
101
+```
102
+
103
+
104
+### Example 2: Silence a particular vLLM logger
105
+
106
+To silence a particular vLLM logger, it is necessary to provide custom logging
107
+configuration for the target logger that configures the logger so that it won't
108
+propagate its log messages to the root vLLM logger.
109
+
110
+When custom configuration is provided for any logger, it is also necessary to
111
+provide configuration for the root vLLM logger since any custom logger
112
+configuration overrides the built-in default logging configuration used by vLLM.
113
+
114
+First, create an appropriate JSON logging configuration file that includes
115
+configuration for the root vLLM logger and for the logger you wish to silence:
116
+
117
+**/path/to/logging_config.json:**
118
+
119
+```json
120
+{
121
+ "formatters": {
122
+ "vllm": {
123
+ "class": "vllm.logging.NewLineFormatter",
124
+ "datefmt": "%m-%d %H:%M:%S",
125
+ "format": "%(levelname)s %(asctime)s %(filename)s:%(lineno)d] %(message)s"
126
+ }
127
+ },
128
+ "handlers": {
129
+ "vllm": {
130
+ "class" : "logging.StreamHandler",
131
+ "formatter": "vllm",
132
+ "level": "INFO",
133
+ "stream": "ext://sys.stdout"
134
+ }
135
+ },
136
+ "loggers": {
137
+ "vllm": {
138
+ "handlers": ["vllm"],
139
+ "level": "DEBUG",
140
+ "propagage": false
141
+ },
142
+ "vllm.example_noisy_logger": {
143
+ "propagate": false
144
+ }
145
+ },
146
+ "version": 1
147
+}
148
+```
149
+
150
+Finally, run vLLM with the `VLLM_LOGGING_CONFIG_PATH` environment variable set
151
+to the path of the custom logging configuration JSON file:
152
+
153
+```bash
154
+VLLM_LOGGING_CONFIG_PATH=/path/to/logging_config.json \
155
+ python3 -m vllm.entrypoints.openai.api_server \
156
+ --max-model-len 2048 \
157
+ --model mistralai/Mistral-7B-v0.1
158
+```
159
+
160
+
161
+### Example 3: Disable vLLM default logging configuration
162
+
163
+To disable vLLM's default logging configuration and silence all vLLM loggers,
164
+simple set `VLLM_CONFIGURE_LOGGING=0` when running vLLM. This will prevent vLLM
165
+for configuring the root vLLM logger, which in turn, silences all other vLLM
166
+loggers.
167
+
168
+```bash
169
+VLLM_CONFIGURE_LOGGING=0 \
170
+ python3 -m vllm.entrypoints.openai.api_server \
171
+ --max-model-len 2048 \
172
+ --model mistralai/Mistral-7B-v0.1
173
+```
174
+
175
+
176
+## Additional resources
177
+
178
+- [`logging.config` Dictionary Schema Details](https://docs.python.org/3/library/logging.config.html#dictionary-schema-details)
tests/test_logger.py
+188
-1
@@ -1,8 +1,19 @@
1
+import json
2
+import logging
3
import os
4
import sys
5
import tempfile
6
+from json.decoder import JSONDecodeError
7
+from tempfile import NamedTemporaryFile
8
+from typing import Any
9
+from unittest.mock import patch
10
+from uuid import uuid4
11
5
-from vllm.logger import enable_trace_function_call
12
+import pytest
13
+
14
+from vllm.logger import (_DATE_FORMAT, _FORMAT, _configure_vllm_root_logger,
15
+ enable_trace_function_call, init_logger)
16
+from vllm.logging import NewLineFormatter
17
18
19
def f1(x):
@@ -25,3 +36,179 @@ def test_trace_function_call():
36
assert "f2" in content
37
sys.settrace(None)
38
os.remove(path)
39
+
40
+
41
+def test_default_vllm_root_logger_configuration():
42
+ """This test presumes that VLLM_CONFIGURE_LOGGING (default: True) and
43
+ VLLM_LOGGING_CONFIG_PATH (default: None) are not configured and default
44
+ behavior is activated."""
45
+ logger = logging.getLogger("vllm")
46
+ assert logger.level == logging.DEBUG
47
+ assert not logger.propagate
48
+
49
+ handler = logger.handlers[0]
50
+ assert handler.stream == sys.stdout
51
+ assert handler.level == logging.INFO
52
+
53
+ formatter = handler.formatter
54
+ assert formatter is not None
55
+ assert isinstance(formatter, NewLineFormatter)
56
+ assert formatter._fmt == _FORMAT
57
+ assert formatter.datefmt == _DATE_FORMAT
58
+
59
+
60
+@patch("vllm.logger.VLLM_CONFIGURE_LOGGING", 1)
61
+@patch("vllm.logger.VLLM_LOGGING_CONFIG_PATH", None)
62
+def test_descendent_loggers_depend_on_and_propagate_logs_to_root_logger():
63
+ """This test presumes that VLLM_CONFIGURE_LOGGING (default: True) and
64
+ VLLM_LOGGING_CONFIG_PATH (default: None) are not configured and default
65
+ behavior is activated."""
66
+ root_logger = logging.getLogger("vllm")
67
+ root_handler = root_logger.handlers[0]
68
+
69
+ unique_name = f"vllm.{uuid4()}"
70
+ logger = init_logger(unique_name)
71
+ assert logger.name == unique_name
72
+ assert logger.level == logging.NOTSET
73
+ assert not logger.handlers
74
+ assert logger.propagate
75
+
76
+ message = "Hello, world!"
77
+ with patch.object(root_handler, "emit") as root_handle_mock:
78
+ logger.info(message)
79
+
80
+ root_handle_mock.assert_called_once()
81
+ _, call_args, _ = root_handle_mock.mock_calls[0]
82
+ log_record = call_args[0]
83
+ assert unique_name == log_record.name
84
+ assert message == log_record.msg
85
+ assert message == log_record.msg
86
+ assert log_record.levelno == logging.INFO
87
+
88
+
89
+@patch("vllm.logger.VLLM_CONFIGURE_LOGGING", 0)
90
+@patch("vllm.logger.VLLM_LOGGING_CONFIG_PATH", None)
91
+def test_logger_configuring_can_be_disabled():
92
+ """This test calls _configure_vllm_root_logger again to test custom logging
93
+ config behavior, however mocks are used to ensure no changes in behavior or
94
+ configuration occur."""
95
+
96
+ with patch("logging.config.dictConfig") as dict_config_mock:
97
+ _configure_vllm_root_logger()
98
+ dict_config_mock.assert_not_called()
99
+
100
+
101
+@patch("vllm.logger.VLLM_CONFIGURE_LOGGING", 1)
102
+@patch(
103
+ "vllm.logger.VLLM_LOGGING_CONFIG_PATH",
104
+ "/if/there/is/a/file/here/then/you/did/this/to/yourself.json",
105
+)
106
+def test_an_error_is_raised_when_custom_logging_config_file_does_not_exist():
107
+ """This test calls _configure_vllm_root_logger again to test custom logging
108
+ config behavior, however it fails before any change in behavior or
109
+ configuration occurs."""
110
+ with pytest.raises(RuntimeError) as ex_info:
111
+ _configure_vllm_root_logger()
112
+ assert ex_info.type == RuntimeError
113
+ assert "File does not exist" in str(ex_info)
114
+
115
+
116
+@patch("vllm.logger.VLLM_CONFIGURE_LOGGING", 1)
117
+def test_an_error_is_raised_when_custom_logging_config_is_invalid_json():
118
+ """This test calls _configure_vllm_root_logger again to test custom logging
119
+ config behavior, however it fails before any change in behavior or
120
+ configuration occurs."""
121
+ with NamedTemporaryFile(encoding="utf-8", mode="w") as logging_config_file:
122
+ logging_config_file.write("---\nloggers: []\nversion: 1")
123
+ logging_config_file.flush()
124
+ with patch("vllm.logger.VLLM_LOGGING_CONFIG_PATH",
125
+ logging_config_file.name):
126
+ with pytest.raises(JSONDecodeError) as ex_info:
127
+ _configure_vllm_root_logger()
128
+ assert ex_info.type == JSONDecodeError
129
+ assert "Expecting value" in str(ex_info)
130
+
131
+
132
+@patch("vllm.logger.VLLM_CONFIGURE_LOGGING", 1)
133
+@pytest.mark.parametrize("unexpected_config", (
134
+ "Invalid string",
135
+ [{
136
+ "version": 1,
137
+ "loggers": []
138
+ }],
139
+ 0,
140
+))
141
+def test_an_error_is_raised_when_custom_logging_config_is_unexpected_json(
142
+ unexpected_config: Any):
143
+ """This test calls _configure_vllm_root_logger again to test custom logging
144
+ config behavior, however it fails before any change in behavior or
145
+ configuration occurs."""
146
+ with NamedTemporaryFile(encoding="utf-8", mode="w") as logging_config_file:
147
+ logging_config_file.write(json.dumps(unexpected_config))
148
+ logging_config_file.flush()
149
+ with patch("vllm.logger.VLLM_LOGGING_CONFIG_PATH",
150
+ logging_config_file.name):
151
+ with pytest.raises(ValueError) as ex_info:
152
+ _configure_vllm_root_logger()
153
+ assert ex_info.type == ValueError
154
+ assert "Invalid logging config. Expected Dict, got" in str(ex_info)
155
+
156
+
157
+@patch("vllm.logger.VLLM_CONFIGURE_LOGGING", 1)
158
+def test_custom_logging_config_is_parsed_and_used_when_provided():
159
+ """This test calls _configure_vllm_root_logger again to test custom logging
160
+ config behavior, however mocks are used to ensure no changes in behavior or
161
+ configuration occur."""
162
+ valid_logging_config = {
163
+ "loggers": {
164
+ "vllm.test_logger.logger": {
165
+ "handlers": [],
166
+ "propagate": False,
167
+ }
168
+ },
169
+ "version": 1
170
+ }
171
+ with NamedTemporaryFile(encoding="utf-8", mode="w") as logging_config_file:
172
+ logging_config_file.write(json.dumps(valid_logging_config))
173
+ logging_config_file.flush()
174
+ with patch("vllm.logger.VLLM_LOGGING_CONFIG_PATH",
175
+ logging_config_file.name), patch(
176
+ "logging.config.dictConfig") as dict_config_mock:
177
+ _configure_vllm_root_logger()
178
+ assert dict_config_mock.called_with(valid_logging_config)
179
+
180
+
181
+@patch("vllm.logger.VLLM_CONFIGURE_LOGGING", 0)
182
+def test_custom_logging_config_causes_an_error_if_configure_logging_is_off():
183
+ """This test calls _configure_vllm_root_logger again to test custom logging
184
+ config behavior, however mocks are used to ensure no changes in behavior or
185
+ configuration occur."""
186
+ valid_logging_config = {
187
+ "loggers": {
188
+ "vllm.test_logger.logger": {
189
+ "handlers": [],
190
+ }
191
+ },
192
+ "version": 1
193
+ }
194
+ with NamedTemporaryFile(encoding="utf-8", mode="w") as logging_config_file:
195
+ logging_config_file.write(json.dumps(valid_logging_config))
196
+ logging_config_file.flush()
197
+ with patch("vllm.logger.VLLM_LOGGING_CONFIG_PATH",
198
+ logging_config_file.name):
199
+ with pytest.raises(RuntimeError) as ex_info:
200
+ _configure_vllm_root_logger()
201
+ assert ex_info.type is RuntimeError
202
+ expected_message_snippet = (
203
+ "VLLM_CONFIGURE_LOGGING evaluated to false, but "
204
+ "VLLM_LOGGING_CONFIG_PATH was given.")
205
+ assert expected_message_snippet in str(ex_info)
206
+
207
+ # Remember! The root logger is assumed to have been configured as
208
+ # though VLLM_CONFIGURE_LOGGING=1 and VLLM_LOGGING_CONFIG_PATH=None.
209
+ root_logger = logging.getLogger("vllm")
210
+ other_logger_name = f"vllm.test_logger.{uuid4()}"
211
+ other_logger = init_logger(other_logger_name)
212
+ assert other_logger.handlers != root_logger.handlers
213
+ assert other_logger.level != root_logger.level
214
+ assert other_logger.propagate
vllm/logger.py
+65
-47
@@ -1,73 +1,91 @@
1
-# Adapted from
2
-# https://github.com/skypilot-org/skypilot/blob/86dc0f6283a335e4aa37b3c10716f90999f48ab6/sky/sky_logging.py
1
"""Logging configuration for vLLM."""
2
import datetime
3
+import json
4
import logging
5
import os
6
import sys
7
from functools import partial
9
-from typing import Optional
8
+from logging import Logger
9
+from logging.config import dictConfig
10
+from os import path
11
+from typing import Dict, Optional
12
13
VLLM_CONFIGURE_LOGGING = int(os.getenv("VLLM_CONFIGURE_LOGGING", "1"))
14
+VLLM_LOGGING_CONFIG_PATH = os.getenv("VLLM_LOGGING_CONFIG_PATH")
15
16
_FORMAT = "%(levelname)s %(asctime)s %(filename)s:%(lineno)d] %(message)s"
17
_DATE_FORMAT = "%m-%d %H:%M:%S"
18
19
+DEFAULT_LOGGING_CONFIG = {
20
+ "formatters": {
21
+ "vllm": {
22
+ "class": "vllm.logging.NewLineFormatter",
23
+ "datefmt": _DATE_FORMAT,
24
+ "format": _FORMAT,
25
+ },
26
+ },
27
+ "handlers": {
28
+ "vllm": {
29
+ "class": "logging.StreamHandler",
30
+ "formatter": "vllm",
31
+ "level": "INFO",
32
+ "stream": "ext://sys.stdout",
33
+ },
34
+ },
35
+ "loggers": {
36
+ "vllm": {
37
+ "handlers": ["vllm"],
38
+ "level": "DEBUG",
39
+ "propagate": False,
40
+ },
41
+ },
42
+ "version": 1,
43
+}
44
+
45
+
46
+def _configure_vllm_root_logger() -> None:
47
+ logging_config: Optional[Dict] = None
48
+
49
+ if not VLLM_CONFIGURE_LOGGING and VLLM_LOGGING_CONFIG_PATH:
50
+ raise RuntimeError(
51
+ "VLLM_CONFIGURE_LOGGING evaluated to false, but "
52
+ "VLLM_LOGGING_CONFIG_PATH was given. VLLM_LOGGING_CONFIG_PATH "
53
+ "implies VLLM_CONFIGURE_LOGGING. Please enable "
54
+ "VLLM_CONFIGURE_LOGGING or unset VLLM_LOGGING_CONFIG_PATH.")
55
17
-class NewLineFormatter(logging.Formatter):
18
- """Adds logging prefix to newlines to align multi-line messages."""
56
+ if VLLM_CONFIGURE_LOGGING:
57
+ logging_config = DEFAULT_LOGGING_CONFIG
58
20
- def __init__(self, fmt, datefmt=None):
21
- logging.Formatter.__init__(self, fmt, datefmt)
59
+ if VLLM_LOGGING_CONFIG_PATH:
60
+ if not path.exists(VLLM_LOGGING_CONFIG_PATH):
61
+ raise RuntimeError(
62
+ "Could not load logging config. File does not exist: %s",
63
+ VLLM_LOGGING_CONFIG_PATH)
64
+ with open(VLLM_LOGGING_CONFIG_PATH, encoding="utf-8",
65
+ mode="r") as file:
66
+ custom_config = json.loads(file.read())
67
23
- def format(self, record):
24
- msg = logging.Formatter.format(self, record)
25
- if record.message != "":
26
- parts = msg.split(record.message)
27
- msg = msg.replace("\n", "\r\n" + parts[0])
28
- return msg
68
+ if not isinstance(custom_config, dict):
69
+ raise ValueError("Invalid logging config. Expected Dict, got %s.",
70
+ type(custom_config).__name__)
71
+ logging_config = custom_config
72
73
+ if logging_config:
74
+ dictConfig(logging_config)
75
31
-_root_logger = logging.getLogger("vllm")
32
-_default_handler: Optional[logging.Handler] = None
76
77
+def init_logger(name: str) -> Logger:
78
+ """The main purpose of this function is to ensure that loggers are
79
+ retrieved in such a way that we can be sure the root vllm logger has
80
+ already been configured."""
81
35
-def _setup_logger():
36
- _root_logger.setLevel(logging.DEBUG)
37
- global _default_handler
38
- if _default_handler is None:
39
- _default_handler = logging.StreamHandler(sys.stdout)
40
- _default_handler.flush = sys.stdout.flush # type: ignore
41
- _default_handler.setLevel(logging.INFO)
42
- _root_logger.addHandler(_default_handler)
43
- fmt = NewLineFormatter(_FORMAT, datefmt=_DATE_FORMAT)
44
- _default_handler.setFormatter(fmt)
45
- # Setting this will avoid the message
46
- # being propagated to the parent logger.
47
- _root_logger.propagate = False
82
+ return logging.getLogger(name)
83
84
50
-# The logger is initialized when the module is imported.
85
+# The root logger is initialized when the module is imported.
86
# This is thread-safe as the module is only imported once,
87
# guaranteed by the Python GIL.
53
-if VLLM_CONFIGURE_LOGGING:
54
- _setup_logger()
55
-
56
-
57
-def init_logger(name: str):
58
- # Use the same settings as above for root logger
59
- logger = logging.getLogger(name)
60
- logger.setLevel(os.getenv("LOG_LEVEL", "DEBUG"))
61
-
62
- if VLLM_CONFIGURE_LOGGING:
63
- if _default_handler is None:
64
- raise ValueError(
65
- "_default_handler is not set up. This should never happen!"
66
- " Please open an issue on Github.")
67
- logger.addHandler(_default_handler)
68
- logger.propagate = False
69
- return logger
70
-
88
+_configure_vllm_root_logger()
89
90
logger = init_logger(__name__)
91
vllm/logging/__init__.py
new
+5
@@ -0,0 +1,5 @@
1
+from vllm.logging.formatter import NewLineFormatter
2
+
3
+__all__ = [
4
+ "NewLineFormatter",
5
+]
vllm/logging/formatter.py
new
+15
@@ -0,0 +1,15 @@
1
+import logging
2
+
3
+
4
+class NewLineFormatter(logging.Formatter):
5
+ """Adds logging prefix to newlines to align multi-line messages."""
6
+
7
+ def __init__(self, fmt, datefmt=None, style="%"):
8
+ logging.Formatter.__init__(self, fmt, datefmt, style)
9
+
10
+ def format(self, record):
11
+ msg = logging.Formatter.format(self, record)
12
+ if record.message != "":
13
+ parts = msg.split(record.message)
14
+ msg = msg.replace("\n", "\r\n" + parts[0])
15
+ return msg