4464 - 数据准备工具包简介：一个用于LLM的可扩展数据准备的Python框架.pdf

1、Session ID:4464Shalisha Witherspoon IBM ResearchIntroduction to Data Prep Kit:A Python framework for scalable data preparation for LLMsIBM TechXchange 2025Open-source Python toolkitlicensed under Apache 2.0,designed for efficient data preparation in LLM workflowsProvides modular transforms and recip

2、es for building and fine-tuning LLMs,including support for RAG and instruct-tuning use casesBuilt for scalable computing environments from local laptops to distributed clusters using tools like Spark and KubeflowSupportscode and language datasets,enabling preprocessing for multilingual corpora,sourc

3、e code repositories,and domain-specific documentsIncludesversatile data connectors for local files,cloud storage(e.g.,S3,IBM COS),and custom connectors,allowing seamless integration with your own data sourcesIntroducing DPKHow to ContributeOpen to all contributors whether youre fixing bugs,improving

4、 docs,or adding new transformsStart with issues labeledgood first issueto get familiar with the codebaseFollow the standard GitHub workflow:fork clone branch commit pull requestEnsure your code adheres to the projects style and testing guidelinesContributions to documentation,examples,and tutorials

5、are equally valuableJoin the community by engaging in discussions,reviews,and feedback loopsNotices and disclaimersIBM,the IBM logo,and are trademarks of International Business Machines Corporation,registered in many jurisdictions worldwide.Other product and service names might be trademarks of IBM

6、or other companies.A current list of IBM trademarks is available on the Web at“Copyright and trademark information”at: comments made in this presentation may be characterized as forward looking under the Private Securities Litigation Reform Act of 1995.Forward-looking statements