扩展生成式人工智能：基础模型的批量推理策略.pdf

1、Scaling Generative AI:Batch Inference Strategies for Foundation ModelsAnkit Mathur&Andrew ShiehWednesday,June 11What can GenAI do for my data?Companies have large amounts of important data and need to process it with generative AI to unlock key insightsContracts&RFPsRegulatory policiesFinancial reco

2、rds&filingsProduct CatalogsClaimsSocial media&forum postsBatch InferenceExtract customersentiments&trendsSummarize findingsBulk personalization of communicationsProduct recommendationsStructured JSON dataTranslating documents intoanother languageUsed for offline analysis in ingestion pipelines or on

3、 historical dataRequires high throughput to achieve low overall per-job latencyTremendous scale of data(terabytes in millions or billions of rows)A lot of compute is needed temporarily,then not at allUsed in interactive surfaces such as chat bots,search results,etc.Requires low per-request latencyRe

4、latively small amounts of data,single articles or short chatsAlways-on provisioned throughputis needed to power productsSimilar but different product requirementsComparing real-time and batch inferenceReal-Time InferenceBatch InferenceREVIEW_IDREVIEWSUMMARIZED_REVIEW123“I love Databricks.However”“Cu

5、stomer loves Databricks”456“AI functions are awesome”“Customer loves AI functions”Setup1.Extract and download data2.Select a model and prompt3.Upload data to batch endpoint4.Wait up to 24 hours5.Ingest results back into governed data storage A naive approach to batch inferenceTedious setup with sign

6、ificant end-to-end latencyREVIEW_IDREVIEW123“I love Databricks.However”456“AI functions are awesome”Data1 dayNaive Batch Inference SystemSELECTreview,ai_query(“databricks-llama-4-maverick”,CONCAT(“Summarize the following user review:”,review)AS summarized_reviewFROMuser_reviewsThe ideal batch infere