@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / dfc77408bd
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 04:15 UTC

[issue templates] add some issue templates (#3412)

youkaichao committed Mar 14, 2024 at 13:16 UTC dfc77408bdca19308cbb28a54dfe697442fbf335
11 files changed +1005
.github/ISSUE_TEMPLATE/100-documentation.yml new
+22
@@ -0,0 +1,22 @@
1 +name: 📚 Documentation
2 +description: Report an issue related to https://docs.vllm.ai/
3 +title: "[Doc]: "
4 +labels: ["doc"]
5 +
6 +body:
7 +- type: textarea
8 + attributes:
9 + label: 📚 The doc issue
10 + description: >
11 + A clear and concise description of what content in https://docs.vllm.ai/ is an issue.
12 + validations:
13 + required: true
14 +- type: textarea
15 + attributes:
16 + label: Suggest a potential alternative/fix
17 + description: >
18 + Tell us how we could improve the documentation in this regard.
19 +- type: markdown
20 + attributes:
21 + value: >
22 + Thanks for contributing 🎉!
.github/ISSUE_TEMPLATE/200-installation.yml new
+39
@@ -0,0 +1,39 @@
1 +name: 🛠️ Installation
2 +description: Report an issue here when you hit errors during installation.
3 +title: "[Installation]: "
4 +labels: ["installation"]
5 +
6 +body:
7 +- type: markdown
8 + attributes:
9 + value: >
10 + #### Before submitting an issue, please make sure the issue hasn't been already addressed by searching through [the existing and past issues](https://github.com/vllm-project/vllm/issues?q=is%3Aissue+sort%3Acreated-desc+).
11 +- type: textarea
12 + attributes:
13 + label: Your current environment
14 + description: |
15 + Please run the following and paste the output below.
16 + ```sh
17 + wget https://raw.githubusercontent.com/vllm-project/vllm/main/collect_env.py
18 + # For security purposes, please feel free to check the contents of collect_env.py before running it.
19 + python collect_env.py
20 + ```
21 + value: |
22 + ```text
23 + The output of `python collect_env.py`
24 + ```
25 + validations:
26 + required: true
27 +- type: textarea
28 + attributes:
29 + label: How you are installing vllm
30 + description: |
31 + Paste the full command you are trying to execute.
32 + value: |
33 + ```sh
34 + pip install -vvv vllm
35 + ```
36 +- type: markdown
37 + attributes:
38 + value: >
39 + Thanks for contributing 🎉!
.github/ISSUE_TEMPLATE/300-usage.yml new
+37
@@ -0,0 +1,37 @@
1 +name: 💻 Usage
2 +description: Raise an issue here if you don't know how to use vllm.
3 +title: "[Usage]: "
4 +labels: ["usage"]
5 +
6 +body:
7 +- type: markdown
8 + attributes:
9 + value: >
10 + #### Before submitting an issue, please make sure the issue hasn't been already addressed by searching through [the existing and past issues](https://github.com/vllm-project/vllm/issues?q=is%3Aissue+sort%3Acreated-desc+).
11 +- type: textarea
12 + attributes:
13 + label: Your current environment
14 + description: |
15 + Please run the following and paste the output below.
16 + ```sh
17 + wget https://raw.githubusercontent.com/vllm-project/vllm/main/collect_env.py
18 + # For security purposes, please feel free to check the contents of collect_env.py before running it.
19 + python collect_env.py
20 + ```
21 + value: |
22 + ```text
23 + The output of `python collect_env.py`
24 + ```
25 + validations:
26 + required: true
27 +- type: textarea
28 + attributes:
29 + label: How would you like to use vllm
30 + description: |
31 + A detailed description of how you want to use vllm.
32 + value: |
33 + I want to run inference of a [specific model](put link here). I don't know how to integrate it with vllm.
34 +- type: markdown
35 + attributes:
36 + value: >
37 + Thanks for contributing 🎉!
.github/ISSUE_TEMPLATE/400-bug report.yml new
+81
@@ -0,0 +1,81 @@
1 +name: 🐛 Bug report
2 +description: Raise an issue here if you find a bug.
3 +title: "[Bug]: "
4 +labels: ["bug"]
5 +
6 +body:
7 +- type: markdown
8 + attributes:
9 + value: >
10 + #### Before submitting an issue, please make sure the issue hasn't been already addressed by searching through [the existing and past issues](https://github.com/vllm-project/vllm/issues?q=is%3Aissue+sort%3Acreated-desc+).
11 +- type: textarea
12 + attributes:
13 + label: Your current environment
14 + description: |
15 + Please run the following and paste the output below.
16 + ```sh
17 + wget https://raw.githubusercontent.com/vllm-project/vllm/main/collect_env.py
18 + # For security purposes, please feel free to check the contents of collect_env.py before running it.
19 + python collect_env.py
20 + ```
21 + value: |
22 + ```text
23 + The output of `python collect_env.py`
24 + ```
25 + validations:
26 + required: true
27 +- type: textarea
28 + attributes:
29 + label: 🐛 Describe the bug
30 + description: |
31 + Please provide a clear and concise description of what the bug is.
32 +
33 + If relevant, add a minimal example so that we can reproduce the error by running the code. It is very important for the snippet to be as succinct (minimal) as possible, so please take time to trim down any irrelevant code to help us debug efficiently. We are going to copy-paste your code and we expect to get the same result as you did: avoid any external data, and include the relevant imports, etc. For example:
34 +
35 + ```python
36 + from vllm import LLM, SamplingParams
37 +
38 + prompts = [
39 + "Hello, my name is",
40 + "The president of the United States is",
41 + "The capital of France is",
42 + "The future of AI is",
43 + ]
44 + sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
45 +
46 + llm = LLM(model="facebook/opt-125m")
47 +
48 + outputs = llm.generate(prompts, sampling_params)
49 +
50 + # Print the outputs.
51 + for output in outputs:
52 + prompt = output.prompt
53 + generated_text = output.outputs[0].text
54 + print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
55 + ```
56 +
57 + If the code is too long (hopefully, it isn't), feel free to put it in a public gist and link it in the issue: https://gist.github.com.
58 +
59 + Please also paste or describe the results you observe instead of the expected results. If you observe an error, please paste the error message including the **full** traceback of the exception. It may be relevant to wrap error messages in ```` ```triple quotes blocks``` ````.
60 + placeholder: |
61 + A clear and concise description of what the bug is.
62 +
63 + ```python
64 + # Sample code to reproduce the problem
65 + ```
66 +
67 + ```
68 + The error message you got, with the full traceback.
69 + ```
70 + validations:
71 + required: true
72 +- type: markdown
73 + attributes:
74 + value: >
75 + ⚠️ Please separate bugs of `transformers` implementation or usage from bugs of `vllm`. If you think anything is wrong with the models' output:
76 +
77 + - Try the counterpart of `transformers` first. If the error appears, please go to [their issues](https://github.com/huggingface/transformers/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc).
78 +
79 + - If the error only appears in vllm, please provide the detailed script of how you run `transformers` and `vllm`, also highlight the difference and what you expect.
80 +
81 + Thanks for contributing 🎉!
.github/ISSUE_TEMPLATE/500-feature request.yml new
+31
@@ -0,0 +1,31 @@
1 +name: 🚀 Feature request
2 +description: Submit a proposal/request for a new vllm feature
3 +title: "[Feature]: "
4 +labels: ["feature"]
5 +
6 +body:
7 +- type: markdown
8 + attributes:
9 + value: >
10 + #### Before submitting an issue, please make sure the issue hasn't been already addressed by searching through [the existing and past issues](https://github.com/vllm-project/vllm/issues?q=is%3Aissue+sort%3Acreated-desc+).
11 +- type: textarea
12 + attributes:
13 + label: 🚀 The feature, motivation and pitch
14 + description: >
15 + A clear and concise description of the feature proposal. Please outline the motivation for the proposal. Is your feature request related to a specific problem? e.g., *"I'm working on X and would like Y to be possible"*. If this is related to another GitHub issue, please link here too.
16 + validations:
17 + required: true
18 +- type: textarea
19 + attributes:
20 + label: Alternatives
21 + description: >
22 + A description of any alternative solutions or features you've considered, if any.
23 +- type: textarea
24 + attributes:
25 + label: Additional context
26 + description: >
27 + Add any other context or screenshots about the feature request.
28 +- type: markdown
29 + attributes:
30 + value: >
31 + Thanks for contributing 🎉!
.github/ISSUE_TEMPLATE/600-new model.yml new
+33
@@ -0,0 +1,33 @@
1 +name: 🤗 Support request for a new model from huggingface
2 +description: Submit a proposal/request for a new model from huggingface
3 +title: "[New Model]: "
4 +labels: ["new model"]
5 +
6 +body:
7 +- type: markdown
8 + attributes:
9 + value: >
10 + #### Before submitting an issue, please make sure the issue hasn't been already addressed by searching through [the existing and past issues](https://github.com/vllm-project/vllm/issues?q=is%3Aissue+sort%3Acreated-desc+).
11 +
12 + #### We also highly recommend you read https://docs.vllm.ai/en/latest/models/adding_model.html first to understand how to add a new model.
13 +- type: textarea
14 + attributes:
15 + label: The model to consider.
16 + description: >
17 + A huggingface url, pointing to the model, e.g. https://huggingface.co/openai-community/gpt2 .
18 + validations:
19 + required: true
20 +- type: textarea
21 + attributes:
22 + label: The closest model vllm already supports.
23 + description: >
24 + Here is the list of models already supported by vllm: https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models . Which model is the most similar to the model you want to add support for?
25 +- type: textarea
26 + attributes:
27 + label: What's your difficulty of supporting the model you want?
28 + description: >
29 + For example, any new operators or new architecture?
30 +- type: markdown
31 + attributes:
32 + value: >
33 + Thanks for contributing 🎉!
.github/ISSUE_TEMPLATE/700-performance discussion.yml new
+51
@@ -0,0 +1,51 @@
1 +name: ⚡ Discussion on the performance of vllm
2 +description: Submit a proposal/discussion about the performance of vllm
3 +title: "[Performance]: "
4 +labels: ["performance"]
5 +
6 +body:
7 +- type: markdown
8 + attributes:
9 + value: >
10 + #### Before submitting an issue, please make sure the issue hasn't been already addressed by searching through [the existing and past issues](https://github.com/vllm-project/vllm/issues?q=is%3Aissue+sort%3Acreated-desc+).
11 +- type: textarea
12 + attributes:
13 + label: Proposal to improve performance
14 + description: >
15 + How do you plan to improve vllm's performance?
16 + validations:
17 + required: false
18 +- type: textarea
19 + attributes:
20 + label: Report of performance regression
21 + description: >
22 + Please provide detailed description of performance comparison to confirm the regression. You may want to run the benchmark script at https://github.com/vllm-project/vllm/tree/main/benchmarks .
23 + validations:
24 + required: false
25 +- type: textarea
26 + attributes:
27 + label: Misc discussion on performance
28 + description: >
29 + Anything about the performance.
30 + validations:
31 + required: false
32 +- type: textarea
33 + attributes:
34 + label: Your current environment (if you think it is necessary)
35 + description: |
36 + Please run the following and paste the output below.
37 + ```sh
38 + wget https://raw.githubusercontent.com/vllm-project/vllm/main/collect_env.py
39 + # For security purposes, please feel free to check the contents of collect_env.py before running it.
40 + python collect_env.py
41 + ```
42 + value: |
43 + ```text
44 + The output of `python collect_env.py`
45 + ```
46 + validations:
47 + required: false
48 +- type: markdown
49 + attributes:
50 + value: >
51 + Thanks for contributing 🎉!
.github/ISSUE_TEMPLATE/800-misc discussion.yml new
+21
@@ -0,0 +1,21 @@
1 +name: 🎲 Misc/random discussions that do not fit into the above categories.
2 +description: Submit a discussion as you like. Note that developers are heavily overloaded and we mainly rely on community users to answer these issues.
3 +title: "[Misc]: "
4 +labels: ["misc"]
5 +
6 +body:
7 +- type: markdown
8 + attributes:
9 + value: >
10 + #### Before submitting an issue, please make sure the issue hasn't been already addressed by searching through [the existing and past issues](https://github.com/vllm-project/vllm/issues?q=is%3Aissue+sort%3Acreated-desc+).
11 +- type: textarea
12 + attributes:
13 + label: Anything you want to discuss about vllm.
14 + description: >
15 + Anything you want to discuss about vllm.
16 + validations:
17 + required: true
18 +- type: markdown
19 + attributes:
20 + value: >
21 + Thanks for contributing 🎉!
.github/ISSUE_TEMPLATE/config.yml new
+1
@@ -0,0 +1 @@
1 +blank_issues_enabled: false
.yapfignore new
+1
@@ -0,0 +1 @@
1 +collect_env.py
collect_env.py new
+688
@@ -0,0 +1,688 @@
1 +# code borrowed from https://github.com/pytorch/pytorch/blob/main/torch/utils/collect_env.py
2 +
3 +# Unlike the rest of the PyTorch this file must be python2 compliant.
4 +# This script outputs relevant system environment info
5 +# Run it with `python collect_env.py` or `python -m torch.utils.collect_env`
6 +import datetime
7 +import locale
8 +import re
9 +import subprocess
10 +import sys
11 +import os
12 +from collections import namedtuple
13 +
14 +
15 +try:
16 + import torch
17 + TORCH_AVAILABLE = True
18 +except (ImportError, NameError, AttributeError, OSError):
19 + TORCH_AVAILABLE = False
20 +
21 +# System Environment Information
22 +SystemEnv = namedtuple('SystemEnv', [
23 + 'torch_version',
24 + 'is_debug_build',
25 + 'cuda_compiled_version',
26 + 'gcc_version',
27 + 'clang_version',
28 + 'cmake_version',
29 + 'os',
30 + 'libc_version',
31 + 'python_version',
32 + 'python_platform',
33 + 'is_cuda_available',
34 + 'cuda_runtime_version',
35 + 'cuda_module_loading',
36 + 'nvidia_driver_version',
37 + 'nvidia_gpu_models',
38 + 'cudnn_version',
39 + 'pip_version', # 'pip' or 'pip3'
40 + 'pip_packages',
41 + 'conda_packages',
42 + 'hip_compiled_version',
43 + 'hip_runtime_version',
44 + 'miopen_runtime_version',
45 + 'caching_allocator_config',
46 + 'is_xnnpack_available',
47 + 'cpu_info',
48 + 'rocm_version', # vllm specific field
49 + 'neuron_sdk_version', # vllm specific field
50 + 'vllm_version', # vllm specific field
51 + 'vllm_build_flags', # vllm specific field
52 + 'gpu_topo', # vllm specific field
53 +])
54 +
55 +DEFAULT_CONDA_PATTERNS = {
56 + "torch",
57 + "numpy",
58 + "cudatoolkit",
59 + "soumith",
60 + "mkl",
61 + "magma",
62 + "triton",
63 + "optree",
64 +}
65 +
66 +DEFAULT_PIP_PATTERNS = {
67 + "torch",
68 + "numpy",
69 + "mypy",
70 + "flake8",
71 + "triton",
72 + "optree",
73 + "onnx",
74 +}
75 +
76 +
77 +def run(command):
78 + """Return (return-code, stdout, stderr)."""
79 + shell = True if type(command) is str else False
80 + p = subprocess.Popen(command, stdout=subprocess.PIPE,
81 + stderr=subprocess.PIPE, shell=shell)
82 + raw_output, raw_err = p.communicate()
83 + rc = p.returncode
84 + if get_platform() == 'win32':
85 + enc = 'oem'
86 + else:
87 + enc = locale.getpreferredencoding()
88 + output = raw_output.decode(enc)
89 + err = raw_err.decode(enc)
90 + return rc, output.strip(), err.strip()
91 +
92 +
93 +def run_and_read_all(run_lambda, command):
94 + """Run command using run_lambda; reads and returns entire output if rc is 0."""
95 + rc, out, _ = run_lambda(command)
96 + if rc != 0:
97 + return None
98 + return out
99 +
100 +
101 +def run_and_parse_first_match(run_lambda, command, regex):
102 + """Run command using run_lambda, returns the first regex match if it exists."""
103 + rc, out, _ = run_lambda(command)
104 + if rc != 0:
105 + return None
106 + match = re.search(regex, out)
107 + if match is None:
108 + return None
109 + return match.group(1)
110 +
111 +def run_and_return_first_line(run_lambda, command):
112 + """Run command using run_lambda and returns first line if output is not empty."""
113 + rc, out, _ = run_lambda(command)
114 + if rc != 0:
115 + return None
116 + return out.split('\n')[0]
117 +
118 +
119 +def get_conda_packages(run_lambda, patterns=None):
120 + if patterns is None:
121 + patterns = DEFAULT_CONDA_PATTERNS
122 + conda = os.environ.get('CONDA_EXE', 'conda')
123 + out = run_and_read_all(run_lambda, "{} list".format(conda))
124 + if out is None:
125 + return out
126 +
127 + return "\n".join(
128 + line
129 + for line in out.splitlines()
130 + if not line.startswith("#")
131 + and any(name in line for name in patterns)
132 + )
133 +
134 +def get_gcc_version(run_lambda):
135 + return run_and_parse_first_match(run_lambda, 'gcc --version', r'gcc (.*)')
136 +
137 +def get_clang_version(run_lambda):
138 + return run_and_parse_first_match(run_lambda, 'clang --version', r'clang version (.*)')
139 +
140 +
141 +def get_cmake_version(run_lambda):
142 + return run_and_parse_first_match(run_lambda, 'cmake --version', r'cmake (.*)')
143 +
144 +
145 +def get_nvidia_driver_version(run_lambda):
146 + if get_platform() == 'darwin':
147 + cmd = 'kextstat | grep -i cuda'
148 + return run_and_parse_first_match(run_lambda, cmd,
149 + r'com[.]nvidia[.]CUDA [(](.*?)[)]')
150 + smi = get_nvidia_smi()
151 + return run_and_parse_first_match(run_lambda, smi, r'Driver Version: (.*?) ')
152 +
153 +
154 +def get_gpu_info(run_lambda):
155 + if get_platform() == 'darwin' or (TORCH_AVAILABLE and hasattr(torch.version, 'hip') and torch.version.hip is not None):
156 + if TORCH_AVAILABLE and torch.cuda.is_available():
157 + if torch.version.hip is not None:
158 + prop = torch.cuda.get_device_properties(0)
159 + if hasattr(prop, "gcnArchName"):
160 + gcnArch = " ({})".format(prop.gcnArchName)
161 + else:
162 + gcnArch = "NoGCNArchNameOnOldPyTorch"
163 + else:
164 + gcnArch = ""
165 + return torch.cuda.get_device_name(None) + gcnArch
166 + return None
167 + smi = get_nvidia_smi()
168 + uuid_regex = re.compile(r' \(UUID: .+?\)')
169 + rc, out, _ = run_lambda(smi + ' -L')
170 + if rc != 0:
171 + return None
172 + # Anonymize GPUs by removing their UUID
173 + return re.sub(uuid_regex, '', out)
174 +
175 +
176 +def get_running_cuda_version(run_lambda):
177 + return run_and_parse_first_match(run_lambda, 'nvcc --version', r'release .+ V(.*)')
178 +
179 +
180 +def get_cudnn_version(run_lambda):
181 + """Return a list of libcudnn.so; it's hard to tell which one is being used."""
182 + if get_platform() == 'win32':
183 + system_root = os.environ.get('SYSTEMROOT', 'C:\\Windows')
184 + cuda_path = os.environ.get('CUDA_PATH', "%CUDA_PATH%")
185 + where_cmd = os.path.join(system_root, 'System32', 'where')
186 + cudnn_cmd = '{} /R "{}\\bin" cudnn*.dll'.format(where_cmd, cuda_path)
187 + elif get_platform() == 'darwin':
188 + # CUDA libraries and drivers can be found in /usr/local/cuda/. See
189 + # https://docs.nvidia.com/cuda/cuda-installation-guide-mac-os-x/index.html#install
190 + # https://docs.nvidia.com/deeplearning/sdk/cudnn-install/index.html#installmac
191 + # Use CUDNN_LIBRARY when cudnn library is installed elsewhere.
192 + cudnn_cmd = 'ls /usr/local/cuda/lib/libcudnn*'
193 + else:
194 + cudnn_cmd = 'ldconfig -p | grep libcudnn | rev | cut -d" " -f1 | rev'
195 + rc, out, _ = run_lambda(cudnn_cmd)
196 + # find will return 1 if there are permission errors or if not found
197 + if len(out) == 0 or (rc != 1 and rc != 0):
198 + l = os.environ.get('CUDNN_LIBRARY')
199 + if l is not None and os.path.isfile(l):
200 + return os.path.realpath(l)
201 + return None
202 + files_set = set()
203 + for fn in out.split('\n'):
204 + fn = os.path.realpath(fn) # eliminate symbolic links
205 + if os.path.isfile(fn):
206 + files_set.add(fn)
207 + if not files_set:
208 + return None
209 + # Alphabetize the result because the order is non-deterministic otherwise
210 + files = sorted(files_set)
211 + if len(files) == 1:
212 + return files[0]
213 + result = '\n'.join(files)
214 + return 'Probably one of the following:\n{}'.format(result)
215 +
216 +
217 +def get_nvidia_smi():
218 + # Note: nvidia-smi is currently available only on Windows and Linux
219 + smi = 'nvidia-smi'
220 + if get_platform() == 'win32':
221 + system_root = os.environ.get('SYSTEMROOT', 'C:\\Windows')
222 + program_files_root = os.environ.get('PROGRAMFILES', 'C:\\Program Files')
223 + legacy_path = os.path.join(program_files_root, 'NVIDIA Corporation', 'NVSMI', smi)
224 + new_path = os.path.join(system_root, 'System32', smi)
225 + smis = [new_path, legacy_path]
226 + for candidate_smi in smis:
227 + if os.path.exists(candidate_smi):
228 + smi = '"{}"'.format(candidate_smi)
229 + break
230 + return smi
231 +
232 +
233 +def get_rocm_version(run_lambda):
234 + """Returns the ROCm version if available, otherwise 'N/A'."""
235 + return run_and_parse_first_match(run_lambda, 'hipcc --version', r'HIP version: (\S+)')
236 +
237 +
238 +def get_neuron_sdk_version(run_lambda):
239 + # Adapted from your install script
240 + try:
241 + result = run_lambda(["neuron-ls"])
242 + return result if result[0] == 0 else 'N/A'
243 + except Exception:
244 + return 'N/A'
245 +
246 +
247 +def get_vllm_version():
248 + try:
249 + import vllm
250 + return vllm.__version__
251 + except ImportError:
252 + return 'N/A'
253 +
254 +
255 +def summarize_vllm_build_flags():
256 + # This could be a static method if the flags are constant, or dynamic if you need to check environment variables, etc.
257 + return 'CUDA Archs: {}; ROCm: {}; Neuron: {}'.format(
258 + os.environ.get('TORCH_CUDA_ARCH_LIST', 'Not Set'),
259 + 'Enabled' if os.environ.get('ROCM_HOME') else 'Disabled',
260 + 'Enabled' if os.environ.get('NEURON_CORES') else 'Disabled',
261 + )
262 +
263 +
264 +def get_gpu_topo(run_lambda):
265 + if get_platform() == 'linux':
266 + return run_and_read_all(run_lambda, 'nvidia-smi topo -m')
267 + return None
268 +
269 +
270 +# example outputs of CPU infos
271 +# * linux
272 +# Architecture: x86_64
273 +# CPU op-mode(s): 32-bit, 64-bit
274 +# Address sizes: 46 bits physical, 48 bits virtual
275 +# Byte Order: Little Endian
276 +# CPU(s): 128
277 +# On-line CPU(s) list: 0-127
278 +# Vendor ID: GenuineIntel
279 +# Model name: Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
280 +# CPU family: 6
281 +# Model: 106
282 +# Thread(s) per core: 2
283 +# Core(s) per socket: 32
284 +# Socket(s): 2
285 +# Stepping: 6
286 +# BogoMIPS: 5799.78
287 +# Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr
288 +# sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl
289 +# xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16
290 +# pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand
291 +# hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced
292 +# fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap
293 +# avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1
294 +# xsaves wbnoinvd ida arat avx512vbmi pku ospke avx512_vbmi2 gfni vaes vpclmulqdq
295 +# avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid md_clear flush_l1d arch_capabilities
296 +# Virtualization features:
297 +# Hypervisor vendor: KVM
298 +# Virtualization type: full
299 +# Caches (sum of all):
300 +# L1d: 3 MiB (64 instances)
301 +# L1i: 2 MiB (64 instances)
302 +# L2: 80 MiB (64 instances)
303 +# L3: 108 MiB (2 instances)
304 +# NUMA:
305 +# NUMA node(s): 2
306 +# NUMA node0 CPU(s): 0-31,64-95
307 +# NUMA node1 CPU(s): 32-63,96-127
308 +# Vulnerabilities:
309 +# Itlb multihit: Not affected
310 +# L1tf: Not affected
311 +# Mds: Not affected
312 +# Meltdown: Not affected
313 +# Mmio stale data: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
314 +# Retbleed: Not affected
315 +# Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
316 +# Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
317 +# Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
318 +# Srbds: Not affected
319 +# Tsx async abort: Not affected
320 +# * win32
321 +# Architecture=9
322 +# CurrentClockSpeed=2900
323 +# DeviceID=CPU0
324 +# Family=179
325 +# L2CacheSize=40960
326 +# L2CacheSpeed=
327 +# Manufacturer=GenuineIntel
328 +# MaxClockSpeed=2900
329 +# Name=Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
330 +# ProcessorType=3
331 +# Revision=27142
332 +#
333 +# Architecture=9
334 +# CurrentClockSpeed=2900
335 +# DeviceID=CPU1
336 +# Family=179
337 +# L2CacheSize=40960
338 +# L2CacheSpeed=
339 +# Manufacturer=GenuineIntel
340 +# MaxClockSpeed=2900
341 +# Name=Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
342 +# ProcessorType=3
343 +# Revision=27142
344 +
345 +def get_cpu_info(run_lambda):
346 + rc, out, err = 0, '', ''
347 + if get_platform() == 'linux':
348 + rc, out, err = run_lambda('lscpu')
349 + elif get_platform() == 'win32':
350 + rc, out, err = run_lambda('wmic cpu get Name,Manufacturer,Family,Architecture,ProcessorType,DeviceID, \
351 + CurrentClockSpeed,MaxClockSpeed,L2CacheSize,L2CacheSpeed,Revision /VALUE')
352 + elif get_platform() == 'darwin':
353 + rc, out, err = run_lambda("sysctl -n machdep.cpu.brand_string")
354 + cpu_info = 'None'
355 + if rc == 0:
356 + cpu_info = out
357 + else:
358 + cpu_info = err
359 + return cpu_info
360 +
361 +
362 +def get_platform():
363 + if sys.platform.startswith('linux'):
364 + return 'linux'
365 + elif sys.platform.startswith('win32'):
366 + return 'win32'
367 + elif sys.platform.startswith('cygwin'):
368 + return 'cygwin'
369 + elif sys.platform.startswith('darwin'):
370 + return 'darwin'
371 + else:
372 + return sys.platform
373 +
374 +
375 +def get_mac_version(run_lambda):
376 + return run_and_parse_first_match(run_lambda, 'sw_vers -productVersion', r'(.*)')
377 +
378 +
379 +def get_windows_version(run_lambda):
380 + system_root = os.environ.get('SYSTEMROOT', 'C:\\Windows')
381 + wmic_cmd = os.path.join(system_root, 'System32', 'Wbem', 'wmic')
382 + findstr_cmd = os.path.join(system_root, 'System32', 'findstr')
383 + return run_and_read_all(run_lambda, '{} os get Caption | {} /v Caption'.format(wmic_cmd, findstr_cmd))
384 +
385 +
386 +def get_lsb_version(run_lambda):
387 + return run_and_parse_first_match(run_lambda, 'lsb_release -a', r'Description:\t(.*)')
388 +
389 +
390 +def check_release_file(run_lambda):
391 + return run_and_parse_first_match(run_lambda, 'cat /etc/*-release',
392 + r'PRETTY_NAME="(.*)"')
393 +
394 +
395 +def get_os(run_lambda):
396 + from platform import machine
397 + platform = get_platform()
398 +
399 + if platform == 'win32' or platform == 'cygwin':
400 + return get_windows_version(run_lambda)
401 +
402 + if platform == 'darwin':
403 + version = get_mac_version(run_lambda)
404 + if version is None:
405 + return None
406 + return 'macOS {} ({})'.format(version, machine())
407 +
408 + if platform == 'linux':
409 + # Ubuntu/Debian based
410 + desc = get_lsb_version(run_lambda)
411 + if desc is not None:
412 + return '{} ({})'.format(desc, machine())
413 +
414 + # Try reading /etc/*-release
415 + desc = check_release_file(run_lambda)
416 + if desc is not None:
417 + return '{} ({})'.format(desc, machine())
418 +
419 + return '{} ({})'.format(platform, machine())
420 +
421 + # Unknown platform
422 + return platform
423 +
424 +
425 +def get_python_platform():
426 + import platform
427 + return platform.platform()
428 +
429 +
430 +def get_libc_version():
431 + import platform
432 + if get_platform() != 'linux':
433 + return 'N/A'
434 + return '-'.join(platform.libc_ver())
435 +
436 +
437 +def get_pip_packages(run_lambda, patterns=None):
438 + """Return `pip list` output. Note: will also find conda-installed pytorch and numpy packages."""
439 + if patterns is None:
440 + patterns = DEFAULT_PIP_PATTERNS
441 +
442 + # People generally have `pip` as `pip` or `pip3`
443 + # But here it is invoked as `python -mpip`
444 + def run_with_pip(pip):
445 + out = run_and_read_all(run_lambda, pip + ["list", "--format=freeze"])
446 + return "\n".join(
447 + line
448 + for line in out.splitlines()
449 + if any(name in line for name in patterns)
450 + )
451 +
452 + pip_version = 'pip3' if sys.version[0] == '3' else 'pip'
453 + out = run_with_pip([sys.executable, '-mpip'])
454 +
455 + return pip_version, out
456 +
457 +
458 +def get_cachingallocator_config():
459 + ca_config = os.environ.get('PYTORCH_CUDA_ALLOC_CONF', '')
460 + return ca_config
461 +
462 +
463 +def get_cuda_module_loading_config():
464 + if TORCH_AVAILABLE and torch.cuda.is_available():
465 + torch.cuda.init()
466 + config = os.environ.get('CUDA_MODULE_LOADING', '')
467 + return config
468 + else:
469 + return "N/A"
470 +
471 +
472 +def is_xnnpack_available():
473 + if TORCH_AVAILABLE:
474 + import torch.backends.xnnpack
475 + return str(torch.backends.xnnpack.enabled) # type: ignore[attr-defined]
476 + else:
477 + return "N/A"
478 +
479 +def get_env_info():
480 + run_lambda = run
481 + pip_version, pip_list_output = get_pip_packages(run_lambda)
482 +
483 + if TORCH_AVAILABLE:
484 + version_str = torch.__version__
485 + debug_mode_str = str(torch.version.debug)
486 + cuda_available_str = str(torch.cuda.is_available())
487 + cuda_version_str = torch.version.cuda
488 + if not hasattr(torch.version, 'hip') or torch.version.hip is None: # cuda version
489 + hip_compiled_version = hip_runtime_version = miopen_runtime_version = 'N/A'
490 + else: # HIP version
491 + def get_version_or_na(cfg, prefix):
492 + _lst = [s.rsplit(None, 1)[-1] for s in cfg if prefix in s]
493 + return _lst[0] if _lst else 'N/A'
494 +
495 + cfg = torch._C._show_config().split('\n')
496 + hip_runtime_version = get_version_or_na(cfg, 'HIP Runtime')
497 + miopen_runtime_version = get_version_or_na(cfg, 'MIOpen')
498 + cuda_version_str = 'N/A'
499 + hip_compiled_version = torch.version.hip
500 + else:
501 + version_str = debug_mode_str = cuda_available_str = cuda_version_str = 'N/A'
502 + hip_compiled_version = hip_runtime_version = miopen_runtime_version = 'N/A'
503 +
504 + sys_version = sys.version.replace("\n", " ")
505 +
506 + conda_packages = get_conda_packages(run_lambda)
507 +
508 + rocm_version = get_rocm_version(run_lambda)
509 + neuron_sdk_version = get_neuron_sdk_version(run_lambda)
510 + vllm_version = get_vllm_version()
511 + vllm_build_flags = summarize_vllm_build_flags()
512 + gpu_topo = get_gpu_topo(run_lambda)
513 +
514 + return SystemEnv(
515 + torch_version=version_str,
516 + is_debug_build=debug_mode_str,
517 + python_version='{} ({}-bit runtime)'.format(sys_version, sys.maxsize.bit_length() + 1),
518 + python_platform=get_python_platform(),
519 + is_cuda_available=cuda_available_str,
520 + cuda_compiled_version=cuda_version_str,
521 + cuda_runtime_version=get_running_cuda_version(run_lambda),
522 + cuda_module_loading=get_cuda_module_loading_config(),
523 + nvidia_gpu_models=get_gpu_info(run_lambda),
524 + nvidia_driver_version=get_nvidia_driver_version(run_lambda),
525 + cudnn_version=get_cudnn_version(run_lambda),
526 + hip_compiled_version=hip_compiled_version,
527 + hip_runtime_version=hip_runtime_version,
528 + miopen_runtime_version=miopen_runtime_version,
529 + pip_version=pip_version,
530 + pip_packages=pip_list_output,
531 + conda_packages=conda_packages,
532 + os=get_os(run_lambda),
533 + libc_version=get_libc_version(),
534 + gcc_version=get_gcc_version(run_lambda),
535 + clang_version=get_clang_version(run_lambda),
536 + cmake_version=get_cmake_version(run_lambda),
537 + caching_allocator_config=get_cachingallocator_config(),
538 + is_xnnpack_available=is_xnnpack_available(),
539 + cpu_info=get_cpu_info(run_lambda),
540 + rocm_version=rocm_version,
541 + neuron_sdk_version=neuron_sdk_version,
542 + vllm_version=vllm_version,
543 + vllm_build_flags=vllm_build_flags,
544 + gpu_topo=gpu_topo,
545 + )
546 +
547 +env_info_fmt = """
548 +PyTorch version: {torch_version}
549 +Is debug build: {is_debug_build}
550 +CUDA used to build PyTorch: {cuda_compiled_version}
551 +ROCM used to build PyTorch: {hip_compiled_version}
552 +
553 +OS: {os}
554 +GCC version: {gcc_version}
555 +Clang version: {clang_version}
556 +CMake version: {cmake_version}
557 +Libc version: {libc_version}
558 +
559 +Python version: {python_version}
560 +Python platform: {python_platform}
561 +Is CUDA available: {is_cuda_available}
562 +CUDA runtime version: {cuda_runtime_version}
563 +CUDA_MODULE_LOADING set to: {cuda_module_loading}
564 +GPU models and configuration: {nvidia_gpu_models}
565 +Nvidia driver version: {nvidia_driver_version}
566 +cuDNN version: {cudnn_version}
567 +HIP runtime version: {hip_runtime_version}
568 +MIOpen runtime version: {miopen_runtime_version}
569 +Is XNNPACK available: {is_xnnpack_available}
570 +
571 +CPU:
572 +{cpu_info}
573 +
574 +Versions of relevant libraries:
575 +{pip_packages}
576 +{conda_packages}
577 +""".strip()
578 +
579 +env_info_fmt += """
580 +ROCM Version: {rocm_version}
581 +Neuron SDK Version: {neuron_sdk_version}
582 +vLLM Version: {vllm_version}
583 +vLLM Build Flags:
584 +{vllm_build_flags}
585 +GPU Topology:
586 +{gpu_topo}
587 +""".strip()
588 +
589 +
590 +def pretty_str(envinfo):
591 + def replace_nones(dct, replacement='Could not collect'):
592 + for key in dct.keys():
593 + if dct[key] is not None:
594 + continue
595 + dct[key] = replacement
596 + return dct
597 +
598 + def replace_bools(dct, true='Yes', false='No'):
599 + for key in dct.keys():
600 + if dct[key] is True:
601 + dct[key] = true
602 + elif dct[key] is False:
603 + dct[key] = false
604 + return dct
605 +
606 + def prepend(text, tag='[prepend]'):
607 + lines = text.split('\n')
608 + updated_lines = [tag + line for line in lines]
609 + return '\n'.join(updated_lines)
610 +
611 + def replace_if_empty(text, replacement='No relevant packages'):
612 + if text is not None and len(text) == 0:
613 + return replacement
614 + return text
615 +
616 + def maybe_start_on_next_line(string):
617 + # If `string` is multiline, prepend a \n to it.
618 + if string is not None and len(string.split('\n')) > 1:
619 + return '\n{}\n'.format(string)
620 + return string
621 +
622 + mutable_dict = envinfo._asdict()
623 +
624 + # If nvidia_gpu_models is multiline, start on the next line
625 + mutable_dict['nvidia_gpu_models'] = \
626 + maybe_start_on_next_line(envinfo.nvidia_gpu_models)
627 +
628 + # If the machine doesn't have CUDA, report some fields as 'No CUDA'
629 + dynamic_cuda_fields = [
630 + 'cuda_runtime_version',
631 + 'nvidia_gpu_models',
632 + 'nvidia_driver_version',
633 + ]
634 + all_cuda_fields = dynamic_cuda_fields + ['cudnn_version']
635 + all_dynamic_cuda_fields_missing = all(
636 + mutable_dict[field] is None for field in dynamic_cuda_fields)
637 + if TORCH_AVAILABLE and not torch.cuda.is_available() and all_dynamic_cuda_fields_missing:
638 + for field in all_cuda_fields:
639 + mutable_dict[field] = 'No CUDA'
640 + if envinfo.cuda_compiled_version is None:
641 + mutable_dict['cuda_compiled_version'] = 'None'
642 +
643 + # Replace True with Yes, False with No
644 + mutable_dict = replace_bools(mutable_dict)
645 +
646 + # Replace all None objects with 'Could not collect'
647 + mutable_dict = replace_nones(mutable_dict)
648 +
649 + # If either of these are '', replace with 'No relevant packages'
650 + mutable_dict['pip_packages'] = replace_if_empty(mutable_dict['pip_packages'])
651 + mutable_dict['conda_packages'] = replace_if_empty(mutable_dict['conda_packages'])
652 +
653 + # Tag conda and pip packages with a prefix
654 + # If they were previously None, they'll show up as ie '[conda] Could not collect'
655 + if mutable_dict['pip_packages']:
656 + mutable_dict['pip_packages'] = prepend(mutable_dict['pip_packages'],
657 + '[{}] '.format(envinfo.pip_version))
658 + if mutable_dict['conda_packages']:
659 + mutable_dict['conda_packages'] = prepend(mutable_dict['conda_packages'],
660 + '[conda] ')
661 + mutable_dict['cpu_info'] = envinfo.cpu_info
662 + return env_info_fmt.format(**mutable_dict)
663 +
664 +
665 +def get_pretty_env_info():
666 + return pretty_str(get_env_info())
667 +
668 +
669 +def main():
670 + print("Collecting environment information...")
671 + output = get_pretty_env_info()
672 + print(output)
673 +
674 + if TORCH_AVAILABLE and hasattr(torch, 'utils') and hasattr(torch.utils, '_crash_handler'):
675 + minidump_dir = torch.utils._crash_handler.DEFAULT_MINIDUMP_DIR
676 + if sys.platform == "linux" and os.path.exists(minidump_dir):
677 + dumps = [os.path.join(minidump_dir, dump) for dump in os.listdir(minidump_dir)]
678 + latest = max(dumps, key=os.path.getctime)
679 + ctime = os.path.getctime(latest)
680 + creation_time = datetime.datetime.fromtimestamp(ctime).strftime('%Y-%m-%d %H:%M:%S')
681 + msg = "\n*** Detected a minidump at {} created on {}, ".format(latest, creation_time) + \
682 + "if this is related to your bug please include it when you file a report ***"
683 + print(msg, file=sys.stderr)
684 +
685 +
686 +
687 +if __name__ == '__main__':
688 + main()