Introduce AI Gateway Features

[CrowleyRajapakse]

Problem

As AI and LLM (Large Language Model) APIs become more common, we face several challenges:

• Control and Governance: Need a unified way to manage AI interactions and ensure quality of service.
• Contextual Information: Passing context to LLM systems to improve response accuracy.
• AI Integration: Enhancing requests and responses with AI capabilities.
• Usage Monitoring: Tracking and summarizing AI API usage for better oversight.

Solution

Need to introduce following functionalities to APK,

• Adding functionality to support AI services as APIs in APK Gateway as CRs. #2424
• Adding Token based Backend RateLimit functionality #2425
• Adding Token based Subscription RateLimit functionality #2426
• Adding APK Conf changes to support adding AI APIs #2427
• Adding AI LLM response specific Analytics metrics #2428
• Support deploying AI APIs from APIM #2429
• Support managing AI APIs created in APK from APIM Control Plane. #2430

Affected Component

None

Version

1.2.0

Implementation

No response

Related Issues

No response

Suggested Labels

No response