| Ask HN: How does one build an LLM router with a labelled set of 100 tasks? | ||
|---|---|---|
| 1 point by zodwick 39 minutes ago │ hide │ past │ favorite │ discuss | ||
| I already spent the time building rubrics for a set of ~100 tasks on various criterias (accuracy, thoroughness communication style ). Then ran these on a few models and now have a labelled-set where I know for each task which is the cheapest/fastest model that gets a minimum threshold ( say 80 ). How does one go on to build a router based on this ? There appears to be no good literature / open source projects on this. I tried a bunch of approaches with the help of claude: - trying to use Jev via a serious of probe Yes/No questions ( does task contain x, does the world have y ...etc ). But this didn't yield good results. - kNN on embeddings of the task - in retrospect this was set to fail from the tasks, as now I think we do need some sort of NN to figure out the world/task complexity. - classifying prompt-complexity - something like https://catalog.ngc.nvidia.com/orgs/nvidia/nemo/models/prompt-task-and-complexity-classifier/- The main issue from a conceptual standpoint is that one cannot fully predict how hard a task is until you are into the task - like you might encounter unknown nuances or issues in later stages, but classification has to happen at start. So I am starting to doubt if this is even possible tbh |
How does one go on to build a router based on this ? There appears to be no good literature / open source projects on this.
I tried a bunch of approaches with the help of claude: - trying to use Jev via a serious of probe Yes/No questions ( does task contain x, does the world have y ...etc ). But this didn't yield good results. - kNN on embeddings of the task - in retrospect this was set to fail from the tasks, as now I think we do need some sort of NN to figure out the world/task complexity. - classifying prompt-complexity - something like https://catalog.ngc.nvidia.com/orgs/nvidia/nemo/models/prompt-task-and-complexity-classifier/-
The main issue from a conceptual standpoint is that one cannot fully predict how hard a task is until you are into the task - like you might encounter unknown nuances or issues in later stages, but classification has to happen at start. So I am starting to doubt if this is even possible tbh