GenAI 开发数据准备的最佳实践.pdf

1、2024 Databricks Inc.All rights reservedBest Practices Best Practices for Data Prep for Data Prep for GenAI for GenAI DevelopmentDevelopmentBrian Kihoon LeeBrian Kihoon Lee12024 Databricks Inc.All rights reservedLilac joined Databricks in March 2024We built LLM-superpowered tools to understand and cu

2、rate text data2Lilac x DatabricksLilac x Databricks2024 Databricks Inc.All rights reserved3A typical RAG data pipelineA typical RAG data pipelineextractcurateindexTodays talk2024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reserved4LIVE DEMO:LIVE DEMO:EXTRACTIONEXTRACTION2024 Da

3、tabricks Inc.All rights reservedPYTHONdef parsed1(filepath):try:doc=pymupdf.open(filepath)#open a documentoutput=n.join(page.get_text()for page in doc)except Exception as e:print(e,filepath)output=return parsed:outputdataset.map(parsed1,input_path=filepath,output_path=parser1,num_jobs=8,execution_ty

4、pe=processes,overwrite=True)Extract data from PDFExtract data from PDF52024 Databricks Inc.All rights reserved6Look at your dataLook at your data2024 Databricks Inc.All rights reservedPYTHONdef parsed2(filepath):try:doc=pymupdf.open(filepath)#open a documenttext_builder=for page in doc:blocks=page.g

5、et_text(blocks)text_builder.append(nn.join(block4.replace(n,)for block in blocks)output=.join(text_builder)except Exception as e:print(e,filepath)output=return parsed:outputdataset.map(parsed2,input_path=filepath,output_path=parser2,num_jobs=8,execution_type=processes)Extract data from PDF,attempt#2

6、Extract data from PDF,attempt#272024 Databricks Inc.All rights reserved8Dont overfit to one documentDont overfit to one document2024 Databricks Inc.All rights reserved9Normalize whitespace and unicodeNormalize whitespace and unicode2024 Databricks Inc.All rights reservedPYTHONfrom unstructured.clean