vllm/.buildkite/lm-eval-harness/configs at v0.18.0 - vllm

DeepSeek-V2-Lite-Chat.yaml

[CI/Build] remove -t for run-lm-eval-gsm-hf-baseline.sh (#16271 )

2025-04-26 18:25:05 +00:00

Meta-Llama-3-8B-Instruct-Channelwise-compressed-tensors.yaml

[CI/Build] remove -t for run-lm-eval-gsm-hf-baseline.sh (#16271 )

2025-04-26 18:25:05 +00:00

Meta-Llama-3-8B-Instruct-FBGEMM-nonuniform.yaml

[CI/Build] remove -t for run-lm-eval-gsm-hf-baseline.sh (#16271 )

2025-04-26 18:25:05 +00:00

Meta-Llama-3-8B-Instruct-FP8-compressed-tensors.yaml

[CI/Build] remove -t for run-lm-eval-gsm-hf-baseline.sh (#16271 )

2025-04-26 18:25:05 +00:00

Meta-Llama-3-8B-Instruct-FP8.yaml

[CI/Build] remove -t for run-lm-eval-gsm-hf-baseline.sh (#16271 )

2025-04-26 18:25:05 +00:00

Meta-Llama-3-8B-Instruct-INT8-compressed-tensors-asym.yaml

[CI/Build] remove -t for run-lm-eval-gsm-hf-baseline.sh (#16271 )

2025-04-26 18:25:05 +00:00

Meta-Llama-3-8B-Instruct-INT8-compressed-tensors.yaml

[CI/Build] remove -t for run-lm-eval-gsm-hf-baseline.sh (#16271 )

2025-04-26 18:25:05 +00:00

Meta-Llama-3-8B-Instruct-nonuniform-compressed-tensors.yaml

[CI/Build] remove -t for run-lm-eval-gsm-hf-baseline.sh (#16271 )

2025-04-26 18:25:05 +00:00

Meta-Llama-3-8B-Instruct.yaml

[CI/Build] remove -t for run-lm-eval-gsm-hf-baseline.sh (#16271 )

2025-04-26 18:25:05 +00:00

Meta-Llama-3-8B-QQQ.yaml

[CI/Build] Add Qwen2.5-VL-7B-Instruct ChartQA Accuracy Tests in CI (#21810 )

2025-10-15 08:09:56 +00:00

Meta-Llama-3-70B-Instruct-FBGEMM-nonuniform.yaml

[CI/Build] remove -t for run-lm-eval-gsm-hf-baseline.sh (#16271 )

2025-04-26 18:25:05 +00:00

Meta-Llama-3-70B-Instruct.yaml

[CI/Build] remove -t for run-lm-eval-gsm-hf-baseline.sh (#16271 )

2025-04-26 18:25:05 +00:00

Meta-Llama-3.2-1B-Instruct-FP8-compressed-tensors.yaml

[CI] Prune down lm-eval small tests (#17012 )

2025-05-08 19:00:26 +00:00

Meta-Llama-3.2-1B-Instruct-INT8-compressed-tensors.yaml

[CI/Build] remove -t for run-lm-eval-gsm-hf-baseline.sh (#16271 )

2025-04-26 18:25:05 +00:00

Meta-Llama-4-Maverick-17B-128E-Instruct-FP8-MM.yaml

[CI/Build] Update Llama4 eval yaml (#27070 )

2025-10-17 04:59:47 +00:00

Meta-Llama-4-Maverick-17B-128E-Instruct-FP8.yaml

[CI/Build][AMD] Add Llama4 Maverick FP8 to AMD CI (#28695 )

2025-12-04 16:07:20 -08:00

Minitron-4B-Base-FP8.yaml

[CI/Build] remove -t for run-lm-eval-gsm-hf-baseline.sh (#16271 )

2025-04-26 18:25:05 +00:00

Mixtral-8x7B-Instruct-v0.1-FP8.yaml

[CI/Build] remove -t for run-lm-eval-gsm-hf-baseline.sh (#16271 )

2025-04-26 18:25:05 +00:00

Mixtral-8x7B-Instruct-v0.1.yaml

[CI/Build] remove -t for run-lm-eval-gsm-hf-baseline.sh (#16271 )

2025-04-26 18:25:05 +00:00

Mixtral-8x22B-Instruct-v0.1-FP8-Dynamic.yaml

[CI/Build] remove -t for run-lm-eval-gsm-hf-baseline.sh (#16271 )

2025-04-26 18:25:05 +00:00

models-large-hopper.txt

Feat/add nemotron nano v3 tests (#33345 )

2026-02-03 08:52:49 -05:00

models-large-rocm.txt

[Rocm][CI] Fix ROCm LM Eval Large Models (8 Card) (#35913 )

2026-03-04 04:50:13 +00:00

models-large.txt

Feat/add nemotron nano v3 tests (#33345 )

2026-02-03 08:52:49 -05:00

models-mm-large-h100.txt

[CI/Build] Add Qwen2.5-VL-7B-Instruct ChartQA Accuracy Tests in CI (#21810 )

2025-10-15 08:09:56 +00:00

models-mm-small.txt

[CI/Build] Add Qwen2.5-VL-7B-Instruct ChartQA Accuracy Tests in CI (#21810 )

2025-10-15 08:09:56 +00:00

models-small-rocm.txt

[Refactor] Make FP8 Linear Ops use kernel abstraction (#27814 )

2026-01-20 14:48:20 +08:00

models-small.txt

[CI] Prune down lm-eval small tests (#17012 )

2025-05-08 19:00:26 +00:00

NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.yaml

Feat/add nemotron nano v3 tests (#33345 )

2026-02-03 08:52:49 -05:00

NVIDIA-Nemotron-3-Nano-30B-A3B-FP8.yaml

Feat/add nemotron nano v3 tests (#33345 )

2026-02-03 08:52:49 -05:00

Qwen1.5-MoE-W4A16-compressed-tensors.yaml

[CI/Build] remove -t for run-lm-eval-gsm-hf-baseline.sh (#16271 )

2025-04-26 18:25:05 +00:00

Qwen2-1.5B-Instruct-FP8W8.yaml

[CI/Build] remove -t for run-lm-eval-gsm-hf-baseline.sh (#16271 )

2025-04-26 18:25:05 +00:00

Qwen2-1.5B-Instruct-INT8-compressed-tensors.yaml

[CI/Build] remove -t for run-lm-eval-gsm-hf-baseline.sh (#16271 )

2025-04-26 18:25:05 +00:00

Qwen2-57B-A14-Instruct.yaml

[CI/Build] remove -t for run-lm-eval-gsm-hf-baseline.sh (#16271 )

2025-04-26 18:25:05 +00:00

Qwen2.5-1.5B-Instruct.yaml

[CI] Prune down lm-eval small tests (#17012 )

2025-05-08 19:00:26 +00:00

Qwen2.5-VL-3B-Instruct-FP8-dynamic.yaml

[CI/Build] Add Qwen2.5-VL-7B-Instruct ChartQA Accuracy Tests in CI (#21810 )

2025-10-15 08:09:56 +00:00

Qwen2.5-VL-7B-Instruct.yaml

[CI/Build] Add Qwen2.5-VL-7B-Instruct ChartQA Accuracy Tests in CI (#21810 )

2025-10-15 08:09:56 +00:00

Qwen3-235B-A22B-Instruct-2507-FP8.yaml

[CI/Build]Add eval config for Qwen3-235B-A22B-Instruct-2507-FP8 (#27113 )

2025-10-30 07:50:56 +00:00

SparseLlama3.1_2of4_fp8_compressed.yaml

[CI/Build] remove -t for run-lm-eval-gsm-hf-baseline.sh (#16271 )

2025-04-26 18:25:05 +00:00