Benjamin Satzger 9116315ec2 Migration: Add limits to inference endpoints table ...

Add three columns to inference_endpoints:

* max_requests
to limit the number of concurrently runnings requests

* max_project_rps
to define how many requests per second we allow a project to make

* max_project_tps
to define how many tokens per second we allow a project to consume

2025-02-11 17:47:12 +01:00

333 B

Raw Permalink Blame History

View Raw