@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / b8afa8b95a
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 01:45 UTC

[MISC] Rework logger to enable pythonic custom logging configuration to be provided (#4273)

Danny Guinther committed May 1, 2024 at 20:34 UTC b8afa8b95a4eee008a9b72440620113e5bfbe962
5 files changed +451 -48
examples/logging_configuration.md new
+178
@@ -0,0 +1,178 @@
1 +# Logging Configuration
2 +
3 +vLLM leverages Python's `logging.config.dictConfig` functionality to enable
4 +robust and flexible configuration of the various loggers used by vLLM.
5 +
6 +vLLM offers two environment variables that can be used to accommodate a range
7 +of logging configurations that range from simple-and-inflexible to
8 +more-complex-and-more-flexible.
9 +
10 +- No vLLM logging (simple and inflexible)
11 + - Set `VLLM_CONFIGURE_LOGGING=0` (leaving `VLLM_LOGGING_CONFIG_PATH` unset)
12 +- vLLM's default logging configuration (simple and inflexible)
13 + - Leave `VLLM_CONFIGURE_LOGGING` unset or set `VLLM_CONFIGURE_LOGGING=1`
14 +- Fine-grained custom logging configuration (more complex, more flexible)
15 + - Leave `VLLM_CONFIGURE_LOGGING` unset or set `VLLM_CONFIGURE_LOGGING=1` and
16 + set `VLLM_LOGGING_CONFIG_PATH=<path-to-logging-config.json>`
17 +
18 +
19 +## Logging Configuration Environment Variables
20 +
21 +### `VLLM_CONFIGURE_LOGGING`
22 +
23 +`VLLM_CONFIGURE_LOGGING` controls whether or not vLLM takes any action to
24 +configure the loggers used by vLLM. This functionality is enabled by default,
25 +but can be disabled by setting `VLLM_CONFIGURE_LOGGING=0` when running vLLM.
26 +
27 +If `VLLM_CONFIGURE_LOGGING` is enabled and no value is given for
28 +`VLLM_LOGGING_CONFIG_PATH`, vLLM will use built-in default configuration to
29 +configure the root vLLM logger. By default, no other vLLM loggers are
30 +configured and, as such, all vLLM loggers defer to the root vLLM logger to make
31 +all logging decisions.
32 +
33 +If `VLLM_CONFIGURE_LOGGING` is disabled and a value is given for
34 +`VLLM_LOGGING_CONFIG_PATH`, an error will occur while starting vLLM.
35 +
36 +### `VLLM_LOGGING_CONFIG_PATH`
37 +
38 +`VLLM_LOGGING_CONFIG_PATH` allows users to specify a path to a JSON file of
39 +alternative, custom logging configuration that will be used instead of vLLM's
40 +built-in default logging configuration. The logging configuration should be
41 +provided in JSON format following the schema specified by Python's [logging
42 +configuration dictionary
43 +schema](https://docs.python.org/3/library/logging.config.html#dictionary-schema-details).
44 +
45 +If `VLLM_LOGGING_CONFIG_PATH` is specified, but `VLLM_CONFIGURE_LOGGING` is
46 +disabled, an error will occur while starting vLLM.
47 +
48 +
49 +## Examples
50 +
51 +### Example 1: Customize vLLM root logger
52 +
53 +For this example, we will customize the vLLM root logger to use
54 +[`python-json-logger`](https://github.com/madzak/python-json-logger) to log to
55 +STDOUT of the console in JSON format with a log level of `INFO`.
56 +
57 +To begin, first, create an appropriate JSON logging configuration file:
58 +
59 +**/path/to/logging_config.json:**
60 +
61 +```json
62 +{
63 + "formatters": {
64 + "json": {
65 + "class": "pythonjsonlogger.jsonlogger.JsonFormatter"
66 + }
67 + },
68 + "handlers": {
69 + "console": {
70 + "class" : "logging.StreamHandler",
71 + "formatter": "json",
72 + "level": "INFO",
73 + "stream": "ext://sys.stdout"
74 + }
75 + },
76 + "loggers": {
77 + "vllm": {
78 + "handlers": ["console"],
79 + "level": "INFO",
80 + "propagate": false
81 + }
82 + },
83 + "version": 1
84 +}
85 +```
86 +
87 +Next, install the `python-json-logger` package if it's not already installed:
88 +
89 +```bash
90 +pip install python-json-logger
91 +```
92 +
93 +Finally, run vLLM with the `VLLM_LOGGING_CONFIG_PATH` environment variable set
94 +to the path of the custom logging configuration JSON file:
95 +
96 +```bash
97 +VLLM_LOGGING_CONFIG_PATH=/path/to/logging_config.json \
98 + python3 -m vllm.entrypoints.openai.api_server \
99 + --max-model-len 2048 \
100 + --model mistralai/Mistral-7B-v0.1
101 +```
102 +
103 +
104 +### Example 2: Silence a particular vLLM logger
105 +
106 +To silence a particular vLLM logger, it is necessary to provide custom logging
107 +configuration for the target logger that configures the logger so that it won't
108 +propagate its log messages to the root vLLM logger.
109 +
110 +When custom configuration is provided for any logger, it is also necessary to
111 +provide configuration for the root vLLM logger since any custom logger
112 +configuration overrides the built-in default logging configuration used by vLLM.
113 +
114 +First, create an appropriate JSON logging configuration file that includes
115 +configuration for the root vLLM logger and for the logger you wish to silence:
116 +
117 +**/path/to/logging_config.json:**
118 +
119 +```json
120 +{
121 + "formatters": {
122 + "vllm": {
123 + "class": "vllm.logging.NewLineFormatter",
124 + "datefmt": "%m-%d %H:%M:%S",
125 + "format": "%(levelname)s %(asctime)s %(filename)s:%(lineno)d] %(message)s"
126 + }
127 + },
128 + "handlers": {
129 + "vllm": {
130 + "class" : "logging.StreamHandler",
131 + "formatter": "vllm",
132 + "level": "INFO",
133 + "stream": "ext://sys.stdout"
134 + }
135 + },
136 + "loggers": {
137 + "vllm": {
138 + "handlers": ["vllm"],
139 + "level": "DEBUG",
140 + "propagage": false
141 + },
142 + "vllm.example_noisy_logger": {
143 + "propagate": false
144 + }
145 + },
146 + "version": 1
147 +}
148 +```
149 +
150 +Finally, run vLLM with the `VLLM_LOGGING_CONFIG_PATH` environment variable set
151 +to the path of the custom logging configuration JSON file:
152 +
153 +```bash
154 +VLLM_LOGGING_CONFIG_PATH=/path/to/logging_config.json \
155 + python3 -m vllm.entrypoints.openai.api_server \
156 + --max-model-len 2048 \
157 + --model mistralai/Mistral-7B-v0.1
158 +```
159 +
160 +
161 +### Example 3: Disable vLLM default logging configuration
162 +
163 +To disable vLLM's default logging configuration and silence all vLLM loggers,
164 +simple set `VLLM_CONFIGURE_LOGGING=0` when running vLLM. This will prevent vLLM
165 +for configuring the root vLLM logger, which in turn, silences all other vLLM
166 +loggers.
167 +
168 +```bash
169 +VLLM_CONFIGURE_LOGGING=0 \
170 + python3 -m vllm.entrypoints.openai.api_server \
171 + --max-model-len 2048 \
172 + --model mistralai/Mistral-7B-v0.1
173 +```
174 +
175 +
176 +## Additional resources
177 +
178 +- [`logging.config` Dictionary Schema Details](https://docs.python.org/3/library/logging.config.html#dictionary-schema-details)
tests/test_logger.py
+188 -1
@@ -1,8 +1,19 @@
1 +import json
2 +import logging
3 import os
4 import sys
5 import tempfile
6 +from json.decoder import JSONDecodeError
7 +from tempfile import NamedTemporaryFile
8 +from typing import Any
9 +from unittest.mock import patch
10 +from uuid import uuid4
11
5 -from vllm.logger import enable_trace_function_call
12 +import pytest
13 +
14 +from vllm.logger import (_DATE_FORMAT, _FORMAT, _configure_vllm_root_logger,
15 + enable_trace_function_call, init_logger)
16 +from vllm.logging import NewLineFormatter
17
18
19 def f1(x):
@@ -25,3 +36,179 @@ def test_trace_function_call():
36 assert "f2" in content
37 sys.settrace(None)
38 os.remove(path)
39 +
40 +
41 +def test_default_vllm_root_logger_configuration():
42 + """This test presumes that VLLM_CONFIGURE_LOGGING (default: True) and
43 + VLLM_LOGGING_CONFIG_PATH (default: None) are not configured and default
44 + behavior is activated."""
45 + logger = logging.getLogger("vllm")
46 + assert logger.level == logging.DEBUG
47 + assert not logger.propagate
48 +
49 + handler = logger.handlers[0]
50 + assert handler.stream == sys.stdout
51 + assert handler.level == logging.INFO
52 +
53 + formatter = handler.formatter
54 + assert formatter is not None
55 + assert isinstance(formatter, NewLineFormatter)
56 + assert formatter._fmt == _FORMAT
57 + assert formatter.datefmt == _DATE_FORMAT
58 +
59 +
60 +@patch("vllm.logger.VLLM_CONFIGURE_LOGGING", 1)
61 +@patch("vllm.logger.VLLM_LOGGING_CONFIG_PATH", None)
62 +def test_descendent_loggers_depend_on_and_propagate_logs_to_root_logger():
63 + """This test presumes that VLLM_CONFIGURE_LOGGING (default: True) and
64 + VLLM_LOGGING_CONFIG_PATH (default: None) are not configured and default
65 + behavior is activated."""
66 + root_logger = logging.getLogger("vllm")
67 + root_handler = root_logger.handlers[0]
68 +
69 + unique_name = f"vllm.{uuid4()}"
70 + logger = init_logger(unique_name)
71 + assert logger.name == unique_name
72 + assert logger.level == logging.NOTSET
73 + assert not logger.handlers
74 + assert logger.propagate
75 +
76 + message = "Hello, world!"
77 + with patch.object(root_handler, "emit") as root_handle_mock:
78 + logger.info(message)
79 +
80 + root_handle_mock.assert_called_once()
81 + _, call_args, _ = root_handle_mock.mock_calls[0]
82 + log_record = call_args[0]
83 + assert unique_name == log_record.name
84 + assert message == log_record.msg
85 + assert message == log_record.msg
86 + assert log_record.levelno == logging.INFO
87 +
88 +
89 +@patch("vllm.logger.VLLM_CONFIGURE_LOGGING", 0)
90 +@patch("vllm.logger.VLLM_LOGGING_CONFIG_PATH", None)
91 +def test_logger_configuring_can_be_disabled():
92 + """This test calls _configure_vllm_root_logger again to test custom logging
93 + config behavior, however mocks are used to ensure no changes in behavior or
94 + configuration occur."""
95 +
96 + with patch("logging.config.dictConfig") as dict_config_mock:
97 + _configure_vllm_root_logger()
98 + dict_config_mock.assert_not_called()
99 +
100 +
101 +@patch("vllm.logger.VLLM_CONFIGURE_LOGGING", 1)
102 +@patch(
103 + "vllm.logger.VLLM_LOGGING_CONFIG_PATH",
104 + "/if/there/is/a/file/here/then/you/did/this/to/yourself.json",
105 +)
106 +def test_an_error_is_raised_when_custom_logging_config_file_does_not_exist():
107 + """This test calls _configure_vllm_root_logger again to test custom logging
108 + config behavior, however it fails before any change in behavior or
109 + configuration occurs."""
110 + with pytest.raises(RuntimeError) as ex_info:
111 + _configure_vllm_root_logger()
112 + assert ex_info.type == RuntimeError
113 + assert "File does not exist" in str(ex_info)
114 +
115 +
116 +@patch("vllm.logger.VLLM_CONFIGURE_LOGGING", 1)
117 +def test_an_error_is_raised_when_custom_logging_config_is_invalid_json():
118 + """This test calls _configure_vllm_root_logger again to test custom logging
119 + config behavior, however it fails before any change in behavior or
120 + configuration occurs."""
121 + with NamedTemporaryFile(encoding="utf-8", mode="w") as logging_config_file:
122 + logging_config_file.write("---\nloggers: []\nversion: 1")
123 + logging_config_file.flush()
124 + with patch("vllm.logger.VLLM_LOGGING_CONFIG_PATH",
125 + logging_config_file.name):
126 + with pytest.raises(JSONDecodeError) as ex_info:
127 + _configure_vllm_root_logger()
128 + assert ex_info.type == JSONDecodeError
129 + assert "Expecting value" in str(ex_info)
130 +
131 +
132 +@patch("vllm.logger.VLLM_CONFIGURE_LOGGING", 1)
133 +@pytest.mark.parametrize("unexpected_config", (
134 + "Invalid string",
135 + [{
136 + "version": 1,
137 + "loggers": []
138 + }],
139 + 0,
140 +))
141 +def test_an_error_is_raised_when_custom_logging_config_is_unexpected_json(
142 + unexpected_config: Any):
143 + """This test calls _configure_vllm_root_logger again to test custom logging
144 + config behavior, however it fails before any change in behavior or
145 + configuration occurs."""
146 + with NamedTemporaryFile(encoding="utf-8", mode="w") as logging_config_file:
147 + logging_config_file.write(json.dumps(unexpected_config))
148 + logging_config_file.flush()
149 + with patch("vllm.logger.VLLM_LOGGING_CONFIG_PATH",
150 + logging_config_file.name):
151 + with pytest.raises(ValueError) as ex_info:
152 + _configure_vllm_root_logger()
153 + assert ex_info.type == ValueError
154 + assert "Invalid logging config. Expected Dict, got" in str(ex_info)
155 +
156 +
157 +@patch("vllm.logger.VLLM_CONFIGURE_LOGGING", 1)
158 +def test_custom_logging_config_is_parsed_and_used_when_provided():
159 + """This test calls _configure_vllm_root_logger again to test custom logging
160 + config behavior, however mocks are used to ensure no changes in behavior or
161 + configuration occur."""
162 + valid_logging_config = {
163 + "loggers": {
164 + "vllm.test_logger.logger": {
165 + "handlers": [],
166 + "propagate": False,
167 + }
168 + },
169 + "version": 1
170 + }
171 + with NamedTemporaryFile(encoding="utf-8", mode="w") as logging_config_file:
172 + logging_config_file.write(json.dumps(valid_logging_config))
173 + logging_config_file.flush()
174 + with patch("vllm.logger.VLLM_LOGGING_CONFIG_PATH",
175 + logging_config_file.name), patch(
176 + "logging.config.dictConfig") as dict_config_mock:
177 + _configure_vllm_root_logger()
178 + assert dict_config_mock.called_with(valid_logging_config)
179 +
180 +
181 +@patch("vllm.logger.VLLM_CONFIGURE_LOGGING", 0)
182 +def test_custom_logging_config_causes_an_error_if_configure_logging_is_off():
183 + """This test calls _configure_vllm_root_logger again to test custom logging
184 + config behavior, however mocks are used to ensure no changes in behavior or
185 + configuration occur."""
186 + valid_logging_config = {
187 + "loggers": {
188 + "vllm.test_logger.logger": {
189 + "handlers": [],
190 + }
191 + },
192 + "version": 1
193 + }
194 + with NamedTemporaryFile(encoding="utf-8", mode="w") as logging_config_file:
195 + logging_config_file.write(json.dumps(valid_logging_config))
196 + logging_config_file.flush()
197 + with patch("vllm.logger.VLLM_LOGGING_CONFIG_PATH",
198 + logging_config_file.name):
199 + with pytest.raises(RuntimeError) as ex_info:
200 + _configure_vllm_root_logger()
201 + assert ex_info.type is RuntimeError
202 + expected_message_snippet = (
203 + "VLLM_CONFIGURE_LOGGING evaluated to false, but "
204 + "VLLM_LOGGING_CONFIG_PATH was given.")
205 + assert expected_message_snippet in str(ex_info)
206 +
207 + # Remember! The root logger is assumed to have been configured as
208 + # though VLLM_CONFIGURE_LOGGING=1 and VLLM_LOGGING_CONFIG_PATH=None.
209 + root_logger = logging.getLogger("vllm")
210 + other_logger_name = f"vllm.test_logger.{uuid4()}"
211 + other_logger = init_logger(other_logger_name)
212 + assert other_logger.handlers != root_logger.handlers
213 + assert other_logger.level != root_logger.level
214 + assert other_logger.propagate
vllm/logger.py
+65 -47
@@ -1,73 +1,91 @@
1 -# Adapted from
2 -# https://github.com/skypilot-org/skypilot/blob/86dc0f6283a335e4aa37b3c10716f90999f48ab6/sky/sky_logging.py
1 """Logging configuration for vLLM."""
2 import datetime
3 +import json
4 import logging
5 import os
6 import sys
7 from functools import partial
9 -from typing import Optional
8 +from logging import Logger
9 +from logging.config import dictConfig
10 +from os import path
11 +from typing import Dict, Optional
12
13 VLLM_CONFIGURE_LOGGING = int(os.getenv("VLLM_CONFIGURE_LOGGING", "1"))
14 +VLLM_LOGGING_CONFIG_PATH = os.getenv("VLLM_LOGGING_CONFIG_PATH")
15
16 _FORMAT = "%(levelname)s %(asctime)s %(filename)s:%(lineno)d] %(message)s"
17 _DATE_FORMAT = "%m-%d %H:%M:%S"
18
19 +DEFAULT_LOGGING_CONFIG = {
20 + "formatters": {
21 + "vllm": {
22 + "class": "vllm.logging.NewLineFormatter",
23 + "datefmt": _DATE_FORMAT,
24 + "format": _FORMAT,
25 + },
26 + },
27 + "handlers": {
28 + "vllm": {
29 + "class": "logging.StreamHandler",
30 + "formatter": "vllm",
31 + "level": "INFO",
32 + "stream": "ext://sys.stdout",
33 + },
34 + },
35 + "loggers": {
36 + "vllm": {
37 + "handlers": ["vllm"],
38 + "level": "DEBUG",
39 + "propagate": False,
40 + },
41 + },
42 + "version": 1,
43 +}
44 +
45 +
46 +def _configure_vllm_root_logger() -> None:
47 + logging_config: Optional[Dict] = None
48 +
49 + if not VLLM_CONFIGURE_LOGGING and VLLM_LOGGING_CONFIG_PATH:
50 + raise RuntimeError(
51 + "VLLM_CONFIGURE_LOGGING evaluated to false, but "
52 + "VLLM_LOGGING_CONFIG_PATH was given. VLLM_LOGGING_CONFIG_PATH "
53 + "implies VLLM_CONFIGURE_LOGGING. Please enable "
54 + "VLLM_CONFIGURE_LOGGING or unset VLLM_LOGGING_CONFIG_PATH.")
55
17 -class NewLineFormatter(logging.Formatter):
18 - """Adds logging prefix to newlines to align multi-line messages."""
56 + if VLLM_CONFIGURE_LOGGING:
57 + logging_config = DEFAULT_LOGGING_CONFIG
58
20 - def __init__(self, fmt, datefmt=None):
21 - logging.Formatter.__init__(self, fmt, datefmt)
59 + if VLLM_LOGGING_CONFIG_PATH:
60 + if not path.exists(VLLM_LOGGING_CONFIG_PATH):
61 + raise RuntimeError(
62 + "Could not load logging config. File does not exist: %s",
63 + VLLM_LOGGING_CONFIG_PATH)
64 + with open(VLLM_LOGGING_CONFIG_PATH, encoding="utf-8",
65 + mode="r") as file:
66 + custom_config = json.loads(file.read())
67
23 - def format(self, record):
24 - msg = logging.Formatter.format(self, record)
25 - if record.message != "":
26 - parts = msg.split(record.message)
27 - msg = msg.replace("\n", "\r\n" + parts[0])
28 - return msg
68 + if not isinstance(custom_config, dict):
69 + raise ValueError("Invalid logging config. Expected Dict, got %s.",
70 + type(custom_config).__name__)
71 + logging_config = custom_config
72
73 + if logging_config:
74 + dictConfig(logging_config)
75
31 -_root_logger = logging.getLogger("vllm")
32 -_default_handler: Optional[logging.Handler] = None
76
77 +def init_logger(name: str) -> Logger:
78 + """The main purpose of this function is to ensure that loggers are
79 + retrieved in such a way that we can be sure the root vllm logger has
80 + already been configured."""
81
35 -def _setup_logger():
36 - _root_logger.setLevel(logging.DEBUG)
37 - global _default_handler
38 - if _default_handler is None:
39 - _default_handler = logging.StreamHandler(sys.stdout)
40 - _default_handler.flush = sys.stdout.flush # type: ignore
41 - _default_handler.setLevel(logging.INFO)
42 - _root_logger.addHandler(_default_handler)
43 - fmt = NewLineFormatter(_FORMAT, datefmt=_DATE_FORMAT)
44 - _default_handler.setFormatter(fmt)
45 - # Setting this will avoid the message
46 - # being propagated to the parent logger.
47 - _root_logger.propagate = False
82 + return logging.getLogger(name)
83
84
50 -# The logger is initialized when the module is imported.
85 +# The root logger is initialized when the module is imported.
86 # This is thread-safe as the module is only imported once,
87 # guaranteed by the Python GIL.
53 -if VLLM_CONFIGURE_LOGGING:
54 - _setup_logger()
55 -
56 -
57 -def init_logger(name: str):
58 - # Use the same settings as above for root logger
59 - logger = logging.getLogger(name)
60 - logger.setLevel(os.getenv("LOG_LEVEL", "DEBUG"))
61 -
62 - if VLLM_CONFIGURE_LOGGING:
63 - if _default_handler is None:
64 - raise ValueError(
65 - "_default_handler is not set up. This should never happen!"
66 - " Please open an issue on Github.")
67 - logger.addHandler(_default_handler)
68 - logger.propagate = False
69 - return logger
70 -
88 +_configure_vllm_root_logger()
89
90 logger = init_logger(__name__)
91
vllm/logging/__init__.py new
+5
@@ -0,0 +1,5 @@
1 +from vllm.logging.formatter import NewLineFormatter
2 +
3 +__all__ = [
4 + "NewLineFormatter",
5 +]
vllm/logging/formatter.py new
+15
@@ -0,0 +1,15 @@
1 +import logging
2 +
3 +
4 +class NewLineFormatter(logging.Formatter):
5 + """Adds logging prefix to newlines to align multi-line messages."""
6 +
7 + def __init__(self, fmt, datefmt=None, style="%"):
8 + logging.Formatter.__init__(self, fmt, datefmt, style)
9 +
10 + def format(self, record):
11 + msg = logging.Formatter.format(self, record)
12 + if record.message != "":
13 + parts = msg.split(record.message)
14 + msg = msg.replace("\n", "\r\n" + parts[0])
15 + return msg