《欧洲数据保护委员会：2025大型语言模型（LLMs）数据保护风险和缓解指南（英文版）（102页）.pdf》由会员分享，可在线阅读，更多相关《欧洲数据保护委员会：2025大型语言模型（LLMs）数据保护风险和缓解指南（英文版）（102页）.pdf（102页珍藏版）》请在三个皮匠报告上搜索。

1、AI Privacy Risks&Mitigations Large Language Models(LLMs)AI Privacy Risks&Mitigations Large Language Models(LLMs)By Isabel BARBER SUPPORT POOL OF EXPERTS PROGRAMME AI Privacy Risks&Mitigations Large Language Models(LLMs)2 As part of the SPE programme,the EDPB may commission contractors to provide rep

2、orts and tools on specific topics.The views expressed in the deliverables are those of their authors and they do not necessarily reflect the official position of the EDPB.The EDPB does not guarantee the accuracy of the information included in the deliverables.Neither the EDPB nor any person acting o

3、n the EDPBs behalf may be held responsible for any use that may be made of the information contained in the deliverables.Some excerpts may be redacted or removed from the deliverables as their publication would undermine the protection of legitimate interests,including,inter alia,the privacy and int

4、egrity of an individual regarding the protection of personal data in accordance with Regulation(EU)2018/1725 and/or the commercial interests of a natural or legal person.Document submitted in February 2025,updated in March 2025 AI Privacy Risks&Mitigations Large Language Models(LLMs)3 TABLE OF CONTE

5、NTS:1.How To Use This Document.4 Structure and Content Overview.4 Guidance for Readers.5 2.Background.6 What Are Large Language Models?.6 How Do Large Language Models Work?.6 Emerging LLM Technologies:The Rise of Agentic AI.12 Common Uses of LLM Systems.15 Performance Measures for LLMs.18 3.Data Flo

6、w and Associated Privacy Risks in LLM Systems.24 The Importance of the AI Lifecycle in Privacy Risk Management.24 Data Flow and Privacy Risks per LLM Service Model.26 Roles in LLMs Service Models According to the AI Act and the GDPR.43 4.Data Protection and Privacy Risk Assessment:Risk Identificatio

7、n.48 Criteria to Consider when Identifying Risks.48 Examples of Privacy Risks in LLM Systems.51 5.Data Protection and Privacy Risk Assessment:Risk Estimation&Evaluation.57 From Risk Identification to Risk Evaluation.57 Criteria to Establish the Probability of Risks in LLM Systems.58 Criteria to Esta

8、blish the Severity of Risks in LLM Systems.60 Risk Evaluation:Classification of Risks.65 6.Data Protection and Privacy Risk Control.66 Risk Treatment Criteria.66 Example of Mitigation Measures Related to Risks of LLM Systems.67 7.Residual Risk Evaluation.74 Identify,Analyze and Evaluate Residual Ris

9、k.74 8.Review&Monitor.75 Risk Management Process Review.75 Continuous Monitoring.75 9.Examples of LLM Systems Risk Assessments.80 First Use Case:A Virtual Assistant(Chatbot)for Customer Queries.80 Second Use Case:LLM System for Monitoring and Supporting Student Progress.91 Third Use Case:AI Assistan

10、t for Travel and Schedule Management.94 10.Reference to Tools,Methodologies,Benchmarks and Guidance.97 Evaluation Metrics for LLMs.97 Other Tools and Guidance.99 Disclaimer by the Author:The examples and references to companies included in this report are provided for illustrative purposes only and

11、do not imply endorsement or suggest that they represent the sole or best options available.While this report strives to provide thorough and insightful information,it is not exhaustive.The technology analysis reflects the state of the art as of March 2025 and is based on extensive research,reference

12、d sources,and the authors expertise.For transparency reasons,the author wants to inform the reader that a LLM system has been used for the exclusive purpose of improving the readability and formatting of parts of the text.AI Privacy Risks&Mitigations Large Language Models(LLMs)4 1.1.How To Use This

13、Document How To Use This Document This document provides practical guidance and tools for developers and users of Large Language Model(LLM)based systems to manage privacy risks associated with these technologies.The risk management methodology outlined in this document is designed to help developers

14、 and users systematically identify,assess,and mitigate privacy and data protection risks,supporting the responsible development and deployment of LLM systems.This guidance also supports the requirements of the GDPR Article 25 Data protection by design and by default and Article 32 Security of proces

15、sing by offering technical and organizational measures to help ensure an appropriate level of security and data protection.However,the guidance is not intended to replace a Data Protection Impact Assessment(DPIA)as required under Article 35 of the GDPR.Instead,it complements the DPIA process by addr

16、essing privacy risks specific to LLM systems,thereby enhancing the robustness of such assessments.Structure and Content Overview Structure and Content Overview The document is structured to guide readers through key technological concepts,the risk management process,main risks and mitigation measure

17、s and practical examples.It aims to support organizations in deploying LLM-based systems responsibly while identifying and mitigating privacy and data protection risks to individuals.Below is an overview of the documents structure and the topics covered in each section:2.Background This section intr

18、oduces Large Language Models,how they work,and their common applications.It also discusses performance evaluation measures,helping readers understand the foundational aspects of LLM systems.3.Data Flow and Associated Privacy Risks in LLM Systems Here,we explore how privacy risks emerge across differ

19、ent LLM service models,emphasizing the importance of understanding data flows throughout the AI lifecycle.This section also identifies risks and mitigations and examines roles and responsibilities under the AI Act and the GDPR.4.Data Protection and Privacy Risk Assessment:Risk Identification This se

20、ction outlines criteria for identifying risks and provides examples of privacy risks specific to LLM systems.Developers and users can use this section as a starting point for identifying risks in their own systems.5.Data Protection and Privacy Risk Assessment:Risk Estimation&Evaluation Guidance on h

21、ow to analyse,classify and assess privacy risks is provided here,with criteria for evaluating both the probability and severity of risks.This section explains how to derive a final risk evaluation to prioritize mitigation efforts effectively.6.Data Protection and Privacy Risk Control This section de

22、tails risk treatment strategies,offering practical mitigation measures for common privacy risks in LLM systems.It also discusses residual risk acceptance and the iterative nature of risk management in AI systems.AI Privacy Risks&Mitigations Large Language Models(LLMs)5 7.Residual Risk Evaluation Eva

23、luating residual risks after mitigation is essential to ensure risks fall within acceptable thresholds and do not require further action.This section outlines how residual risks are evaluated to determine whether additional mitigation is needed or if the model or LLM system is ready for deployment.8

24、.Review&Monitor This section covers the importance of reviewing risk management activities and maintaining a risk register.It also highlights the importance of continuous monitoring to detect emerging risks,assess real-world impact,and refine mitigation strategies.9.Examples of LLM Systems Risk Asse

25、ssments Three detailed use cases are provided to demonstrate the application of the risk management framework in real-world scenarios.These examples illustrate how risks can be identified,assessed,and mitigated across various contexts.10.Reference to Tools,Methodologies,Benchmarks,and Guidance The f

26、inal section compiles tools,evaluation metrics,benchmarks,methodologies,and standards to support developers and users in managing risks and evaluating the performance of LLM systems.Guidance for Readers Guidance for Readers For Developers:Use this guidance to integrate privacy risk management into t

27、he development lifecycle and deployment of your LLM based systems,from understanding data flows to how to implement risk identification and mitigation measures.For Users:Refer to this document to evaluate the privacy risks associated with LLM systems you plan to deploy and use,helping you adopt resp

28、onsible practices and protect individuals privacy.For Decision-makers:The structured methodology and use case examples will help you assess the compliance of LLM systems and make informed risk-based decisions.AI Privacy Risks&Mitigations Large Language Models(LLMs)6 2.2.Background Background What Ar

29、e Large Language Models?What Are Large Language Models?Large Language Models(LLMs)represent a transformative advancement in artificial intelligence.These general purpose models are trained on extensive datasets,which often encompass publicly available content,proprietary datasets,and specialized dom

30、ain-specific data.Their applications are diverse,ranging from text generation and summarization to coding assistance,sentiment analysis,and more.Some LLMs are multimodal LLMs,capable of processing and generating multiple data modalities such as image,audio or video.The development of LLMs has been m

31、arked by key technological milestones that have shaped their evolution.Early advancements in the 1960s and 1970s included rule-based systems like ELIZA,which laid foundational principles for simulating human conversation through predefined patterns.In 2017,the introduction of transformer architectur

32、es(see Figure 2)in the seminal paper Attention Is All You Need1 revolutionized the field by enabling efficient handling of contextual relationships within text sequences.Subsequent developments,such as OpenAIs GPT series and Googles BERT(see Figure 3),have set benchmarks for natural language process

33、ing(NLP)2,culminating in models like GPT-4,LaMDA3,and DeepSeek-V34(see Figure 4)integrating multimodal capabilities.How Do Large Language Models Work?How Do Large Language Models Work?LLMs are advanced deep learning models designed to process and generate human-like language.They rely on the transfo

34、rmer architecture5,which uses attention mechanisms to understand context and relationships between words.While most state of the art LLMs rely on transformers due to their scalability and effectiveness,alternatives6 exist based on RNN(Recurring Neural Networks)such as LSTM(Long-short Term Memory)and

35、 others that are actively being researched7.For now,transformers dominate general-purpose language models,but innovations in architectures such as those introduced by DeepSeeks models,may reshape the landscape in the future.The development8of LLMs can be divided into several key stages:1.Training Ph

36、ase:Building the Model 1.Training Phase:Building the Model In this phase LLMs learn patterns,context,and structure in language by analyzing vast datasets.1.Dataset Collection:The foundation of LLM training lies in the use of extensive datasets(such as such as Common Crawl and Wikipedia)that are care

37、fully curated to ensure they are relevant,diverse,and high-quality.Filtering eliminates low-quality or redundant content,aligning the training data with the intended goals of the model.2.Data Pre-processing:1 A.Vaswan et al.,Attention Is All You Need(2023)https:/arxiv.org/pdf/1706.03762 2 Wikipedia,

38、Natural language processing(2025)https:/en.wikipedia.org/wiki/Natural_language_processing 3 E.Collins and Z.Ghahramani,LaMDA:our breakthrough conversation technology(2021)https:/blog.google/technology/ai/lamda/4 Github,Deepseek(n.d)https:/ 5 Wikipedia,Deep Learning Architecture(2025)https:/en.wikipe

39、dia.org/wiki/Transformer_(deep_learning_architecture)6 Artificial Intelligence,Why does the transformer do better than RNN and LSTM in long-range context dependencies?(2020)https:/ 7 A.Gu,T.Dao,Mamba:Linear-Time Sequence Modeling with Selective State Spaces(2024)https:/arxiv.org/pdf/2312.00752,B.Pen

40、g et al,RWKV:Reinventing RNNs for the Transformer Era(2023)https:/arxiv.org/pdf/2305.13048 8 Y.Liu et al.,Understanding LLMs:A Comprehensive Overview from Training to Inference(2024)https:/arxiv.org/pdf/2401.02038v2 AI Privacy Risks&Mitigations Large Language Models(LLMs)7 Text is cleaned and normal

41、ized by removing inconsistencies(e.g.,special characters)and irrelevant content,ensuring uniformity in the training data.Text data is broken into smaller units called tokens,which can be words,subwords,or even individual characters.Tokenization algorithms transforms unstructured text into manageable

42、 sequences for computational processing.Tokens are converted into numerical IDs that represent their vocabulary position.These IDs are then transformed into word embeddings9dense vector representations that capture semantic similarities and relationships between words.For instance,semantically relat

43、ed words like“king”and“queen”will occupy nearby positions in the embedding space.Figure 1.Source:S.Anala A Guide to Word Embedding(2020)https:/ 3.Transformer Architecture:10 Transformer architectures can be categorized into three main types:encoder-only,encoder-decoder,and decoder-only.While encoder

44、-only architectures were foundational in earlier models,they are generally not used in the latest generation of LLMs.Most state of the art LLMs today use decoder-only architectures,while encoder-decoder models are still used in tasks like translation and instruction tuning.Encoder:11 The encoder tak

45、es the input text and converts it into a contextualized representation by analyzing relationships between words.Key elements include:o Token embeddings:Tokens are transformed into numerical vectors that capture their meaning.o Positional encodings:Since the transformer processes words in parallel,po

46、sitional encodings are added to token embeddings to represent the order of words,preserving the structure of the input.o Attention mechanisms:The encoder evaluates the importance of each word relative to others in the input sequence,capturing dependencies and context.For example,it helps distinguish

47、 between“park”as a verb and“park”as a location based on the surrounding text.o Feed-Forward Network:A series of transformations are applied to refine the contextualized word representations,preparing them for subsequent stages.9 V.Zhukov,A Guide to Understanding Word Embeddings in Natural Language P

48、rocessing(NLP)(2023)https:/ingestai.io/blog/word-embeddings-in-nlp 10 See footnote 1 11 Geeksforgeels,Architecture and Working of Transformers in Deep Learning(2025)https:/www.geeksforgeeks.org/architecture-and-working-of-transformers-in-deep-learning/AI Privacy Risks&Mitigations Large Language Mode

49、ls(LLMs)8 Decoder:12 The decoder generates text by predicting one token at a time.It builds upon the encoders output(if used)and the sequence of tokens already generated.Key elements include:o Input:Combines encoder outputs with tokens generated so far.o Attention mechanisms:13 Ensures each token co

50、nsiders previously generated tokens to maintain coherence and context.o Feed-Forward Network(FFN):14 This layer refines the token representations to ensure they are relevant and coherent.o Masked attention:During training,future tokens are hidden from the model,ensuring it predicts outputs step by s

51、tep without cheating.Figure 2.Transformer architecture.Source:Vaswani et al.Attention Is All You Need (2023)https:/arxiv.org/pdf/1706.03762 Figure 3.A comparison of the architectures for the Transformer,GPT and BERT.Source:B.Smith A Complete Guide to BERT with Code(2024)https:/ 12 idem 13 The archit

52、ecture of DeepSeek models contains an innovative attention mechanism called Multi-head Latent Attention(MLA)that compresses Key/Value vectors offering better compute and memory efficiency.14 DeepSeek models employ the DeepSeekMoE architecture based on Mixture-of-Experts(MoE)introducing multiple para

53、llel expert networks(FFNs)instead of a single FFN.AI Privacy Risks&Mitigations Large Language Models(LLMs)9 Mixture of Experts(MoE)is a technique used to improve transformer-based LLMs making them more efficient and scalable.Instead of using the entire model for every input,MoE activates only a few

54、smaller parts of the modelcalled expertsbased on what the input needs.This means the model can be much larger overall,but only the necessary parts are used at any time,saving computing power without losing performance Figure 4.Illustration of DeepSeek-V3s basic architecture called DeepSeekMoE based

55、on Mixture-of-Experts(MoE).Source:DeepSeek-V3 Technical Report https:/arxiv.org/pdf/2412.19437 4.Training/Feedback loop&Optimization15 The training phase of LLMs relies on a structured optimization loop to enhance the models ability to generate accurate outputs.This iterative process consists of the

56、 following steps:Loss calculation:After generating an output sequence,the model compares it to the target sequence(the correct answer).A loss function quantifies the error,providing a numerical measure of how far the predicted output deviates from the desired result(the loss).Backward pass:The obtai

57、ned loss value is used to compute gradients,which indicate how much each model parameter(e.g.,weights and biases)contributed to the error.These gradients highlight areas where the model needs improvement.Parameter update:Using an optimization algorithm,such as Adam or SGD(Stochastic Gradient Descent

58、),the models parameters are adjusted.This step reduces the error for future predictions by refining the internal model weights.Repetition:This process repeats for thousands or millions of iterations.Each cycle incrementally improves the models performance.Training stops when the model reaches a bala

59、nce between accuracy on training data and generalization to unseen inputs.2.Continuous Improvement Model Alignment(post training)2.Continuous Improvement Model Alignment(post training)Pre-trained models,while powerful,are generally not immediately useful in their raw form.To make models behaviour al

60、ign with ethical considerations and user preferences16 they need to be tuned.This 15 PyTorch Loss.backward()and Optimizer.step():A Deep Dive for Machine Learning(2025)https:/iifx.dev/en/articles/315715245 16 C.R.Wolfe,Understanding and Using Supervised Fine-Tuning(SFT)for Language Models (2023)https

61、:/ AI Privacy Risks&Mitigations Large Language Models(LLMs)10 process often involves the use of techniques such as supervised fine-tuning on domain-specific data or Reinforcement Learning with Human Feedback(RLHF).The most common alignment methods are:Supervised Fine-Tuning(SFT):17 This approach inv

62、olves training a pre-trained model on a labeled dataset tailored to a specific task,with adjustments made to some or all of its parameters to enhance performance for that task.Instruction Tuning:18 This technique is used to optimize the LLM for following user instructions and handling conversational

63、 tasks.Reinforcement Learning with Human Feedback(RLHF):19 This method uses human feedback to train a reward model(RM),which helps guide the AI during its learning process.The reward model acts as a scorekeeper,showing the AI how well its performing based on the feedback.Techniques like Proximal Pol

64、icy Optimization(PPO)are then used to fine-tune the language model.In simple terms,the language model learns to make better decisions based on the reward signals it receives.Direct Preference Optimization(DPO)20 is an emerging reinforcement learning approach that simplifies this process by directly

65、incorporating user preference data into the models optimization process.While RLHF aims to align the model with human preferences across diverse scenarios using human feedback,another variation of the PPO technique called Group Relative Policy Optimization(GRPO)21 introduced by DeepSeek researchers,

66、takes a different approach.Instead of relying on human annotations,GRPO uses computer-generated scores to guide the models learning process and reasoning capabilities in an automated manner.Parameter-Efficient Fine-Tuning(PEFT):22 This technique adapts pre-trained models to new tasks by training onl

67、y some of the models parameters,leaving the majority of the pre-trained model unchanged.Some PEFT techniques are adapters,LoRA,QLoRA and prompt-tuning.Retrieval-Augmented Generation(RAG):232425 This method enhances LLMs by integrating information retrieval capabilities,enabling them to reference spe

68、cific documents.This approach allows LLMs to incorporate domain-specific or updated information when responding to user queries.Transfer Learning:2627 With this technique,knowledge learned from a task is re-used in another model.Feedback loops:28 Real-world user feedback helps refine the models beha

69、vior,allowing it to adapt to new contexts or correct inaccuracies.Feedback can be collected through user behaviour,for instance inferring whether the user engages with or ignores a response.Feedback can also be collected when users directly provide feedback on the models output,such as a thumbs-up/t

70、humbs-down rating,qualitative comments,or error corrections.The LLM is then refined based on this feedback.17 Bergmann,D.What IS fine-tuning?(2024)https:/ 18 D.Bergman,What is instruction tuning?(2024)https:/ 19 S.Chaudhari et al.RLHF Deciphered:A Critical Analysis of Reinforcement Learning from Hum

71、an Feedback for LLMs(2024)https:/arxiv.org/abs/2404.08555 20 R.Rafailov,Direct Preference Optimization:Your Language Model is Secretly a Reward Model(2024)https:/arxiv.org/abs/2305.18290 21 Z.Shao,DeepSeekMath:Pushing the Limits of Mathematical Reasoning in Open Language Models(2024)https:/arxiv.org

72、/abs/2402.03300 22 Stryker,C.et al.,What is parameter-efficient fine-tuning(PEFT)?(2024)https:/ 23 AWS,What is RAG(Retrieval-Augmented Generation)?(2025)https:/ Wikipedia,Retrieval Augmented Generation(2025)https:/en.wikipedia.org/wiki/Retrieval-augmented_generation 25 IBM,Retrieval Augmented Genera

73、tion(2025)https:/ 26 V.Chaba,Understanding the Differences:Fine-Tuning vs.Transfer Learning (2023)https:/dev.to/luxacademy/understanding-the-differences-fine-tuning-vs-transfer-learning-370 27 Wikipedia,Transfer Learning(2025)https:/en.wikipedia.org/wiki/Transfer_learning 28 Nebuly AI,LLM Feedback L

74、oop(2024)https:/ AI Privacy Risks&Mitigations Large Language Models(LLMs)11 3.Inference Phase:Generating Outputs 3.Inference Phase:Generating Outputs Once trained,the model enters the inference phase,where it generates outputs based on new inputs following these steps:1.Input:The users query is proc

75、essed through tokenization and embedding,transforming it into a format the model can understand.2.Processing:The input passes through the transformer architecture,where attention mechanisms and decoder layers predict the next tokens in the sequence.The decoder produces a vector of scores(called logi

76、ts)for each word in the vocabulary.These scores are then passed through the Softmax29 function,which converts them into probabilities.The model selects the most probable token as the next word in the sequence,ensuring that the generated text is coherent and contextually relevant.3.Output:The model p

77、roduces probabilities for potential next words,selecting the most likely options based on the input and context.These predictions are combined to generate coherent and relevant responses.The three key stages described outline how a traditional text-only LLM is developed.Multimodal LLMs follow a simi

78、lar process but to handle multiple data modalities,they incorporate specialized components such as modality-specific encoders,connectors and cross-modal fusion mechanisms to integrate the different data representations,along with a shared decoder to generate coherent outputs across modalities.Their

79、development also involves pre-training and fine-tuning stages;however,some architectures build multimodal LLMs by fine-tuning an already pre-trained text-only LLM rather than training one from scratch.Figure 5.Typical Multimodal LLM(MLLM)architecture.Source:Y.Shukang et al.A Survey on Multimodal Lar

80、ge Language Models(2024)https:/arxiv.org/abs/2306.13549 In practice,LLMs are often part of a system and can be accessed directly via APIs,are embedded within SaaS platforms,deployed as off-the-shelf foundational models fine-tuned for specific use cases,or integrated into on-premise solutions.It is i

81、mportant to note that while LLMs are essential components of AI systems,they do not constitute AI systems on their own.For an LLM to become part of an AI system,additional components such as a user interface,must be integrated to enable it to function as a complete system.30Throughout this document,

82、we will refer to such complete systems as LLM-based systems or simply LLM systems to emphasize their broader context and functionality.This distinction is crucial when assessing the risks associated with these systems,as an LLM system inherently carries more risks due to its additional components an

83、d integrations compared to a standalone LLM.29 Wikipedia,Softmax Function(2025)https:/en.wikipedia.org/wiki/Softmax_function 30 Recital 97 AI Act AI Privacy Risks&Mitigations Large Language Models(LLMs)12 Each stage of an LLMs development lifecycle could introduce potential privacy risks,as the mode

84、l interacts with large datasets that might contain personal data and it generates outputs based on that data.Some of the key privacy concerns may occur during:The collection of data:The training,testing and validation set could contain identifiable personal data,sensitive data or special category of

85、 data.Inference:Generated outputs could inadvertently reveal private information or contain misinformation.RAG process:We might use knowledge bases containing sensitive data or identifiable personal data without implementing proper safeguards.Feedback loops:User interactions might be stored without

86、adequate safeguards.Emerging LLM Technologies:The Rise of Agentic AI Emerging LLM Technologies:The Rise of Agentic AI According to a recent report from Deloitte,31 by 2027,50%of companies leveraging generative AI are expected to have launched pilots or proofs of concept to implement agentic AI syste

87、ms.These systems are envisioned to function as intelligent assistants,capable of autonomously managing complex tasks with minimal human supervision.AI Agents32 are autonomous systems that can be built on top of LLMs and that can perform complex tasks by combining the capabilities of LLMs with reason

88、ing,decision-making,and interaction capabilities.AI agents are proactive,capable of goal-oriented behavior such as planning,executing tasks,and iterating based on feedback.They can operate independently and are designed to achieve specific objectives by orchestrating multiple actions in sequence.The

89、y can also incorporate feedback to refine their actions or responses over time.Advanced AI agents may integrate capabilities from other AI systems,such as computer vision or audio processing,to handle diverse data inputs.33 The concept of agentic AI remains an evolving and not yet fully defined doma

90、in.Different organizations and researchers propose varying interpretations of what constitutes an agentic AI system.For example,at Anthropic34,they emphasize a significant architectural distinction between workflows and agents:Workflows are structured systems where LLMs and tools operate in a predef

91、ined manner,following orchestrated code paths.Agents,in contrast,are designed to function dynamically.They allow LLMs to autonomously direct their processes and determine how to use tools and resources to achieve objectives.An Overview of AI Agents and their ArchitectureAn Overview of AI Agents and

92、their Architecture3535 In systems powered by LLMs,the LLM serves as a central brain providing the foundational abilities for natural language understanding and reasoning.This ability is augmented with additional components that equip the agent to plan,learn,and interact dynamically with its environm

93、ent,enabling it to handle tasks that go beyond standalone LLM capabilities.The architecture of an AI agent focuses on critical components that work together to enable sophisticated behavior and adaptability in real-world scenarios.The architecture is modular,involving distinct components for percept

94、ion,reasoning,planning,memory management,and action.This 31 J.Loucks Autonomous generative AI agents:Under development(2024)https:/ 32 C.Gadelho,Building AI and LLM Agents from the Ground Up:A Step-by-Step Guide(2024)https:/www.tensorops.ai/post/building-ai-and-llm-agents-from-the-ground-up-a-step-b

95、y-step-guide 33 OpenAIs Operator(2025)https:/ Anthropic,Building effective agents(2024)https:/ 35 idem AI Privacy Risks&Mitigations Large Language Models(LLMs)13 modularity allows the system to handle complex tasks,interact dynamically with their environment,and refine performance iteratively.Some o

96、f the most common modules currently used are:1.Perception module This module handles the agents ability to process inputs from the environment and format them into a structure that the LLM can understand.It converts raw inputs(e.g.,text,voice,or data streams)into embeddings or structured formats tha

97、t can be processed by the reasoning module.2.Reasoning module The reasoning module enables the agent to interpret input data,analyze its context,and decompose complex tasks into smaller,manageable subtasks.It leverages the LLMs ability to understand and process natural language to make decisions.The

98、 reasoning mechanism enables the agent to analyze user inputs to determine the best course of action and leverage the appropriate tool or resource to achieve the desired outcome.3.Planning module The planning module determines how the agent will execute the subtasks identified by the reasoning modul

99、e.It organizes and sequences actions to achieve a defined goal.4.Memory and state management To maintain context and continuity,the agent keeps track of past interactions.Memory allows the AI agent to store and retrieve context,both within a single interaction and across multiple sessions.o Short-Te

100、rm Memory:Maintains context within the current interaction to ensure coherence in responses.o Long-Term Memory:Stores user preferences,past interactions,and learned insights for personalization.5.Action module This module is responsible for executing the plan and interacting with the external enviro

101、nment.It carries out the tasks identified and planned by earlier modules.The agent must have access to a defined set of tools,such as APIs,databases,or external systems,which it can use to accomplish the specific tasks.For example,an AI assistant might use a calendar API for scheduling or a booking

102、service for travel reservations.6.Feedback and iteration loop The feedback loop enables the agent to evaluate the success of its actions and adjust its behavior dynamically.It incorporates user corrections,system logs,and performance metrics to refine reasoning,planning,and execution over time.Inter

103、action Between AI Agent,Memory,and Environment Interaction Between AI Agent,Memory,and Environment The agent interacts continuously with its memory and external environment.Context from memory enhances task relevance and continuity while external data(e.g.,user queries,sensor inputs)drives decision-

104、making and task execution.AI Privacy Risks&Mitigations Large Language Models(LLMs)14 Figure 6.Source:Z.Deng et al.AI Agents Under Threat:A Survey of Key Security Challenges and Future Pathways(2024)https:/ Small Language Models(SLMs)and their role in AI AgentsSmall Language Models(SLMs)and their rol

105、e in AI Agents3636 Small Language Models(SLMs)are lightweight,task-specific models designed to handle simpler or more focused tasks compared to Large Language Models(LLMs).While LLMs excel at understanding and generating complex language,SLMs are optimized for specific applications,such as text clas

106、sification,entity recognition,or sentiment analysis.SLMs can complement LLMs in agentic AI by taking on specialized tasks that do not require the extensive computational resources or generality of LLMs.In AI agents,SLMs can enhance efficiency and privacy by processing data locally,reducing reliance

107、on centralized LLMs.This modular approach allows agents to allocate tasks improving overall performance and security.To fully leverage the capabilities of LLMs within organizations,it is essential to adapt the models to the organizations specific knowledge base and business processes.This customizat

108、ion,often achieved by fine-tuning the LLM with organization-specific data,can result in a domain-focused small language model(SLM).37 Model OrchestrationModel Orchestration3838 For agentic AI to seamlessly integrate the strengths of both SLMs39 and LLMs,a system is needed to dynamically manage which

109、 model handles which task.This is where model orchestration plays a critical role,ensuring efficient and secure collaboration between different models.In agentic AI,orchestration determines the most appropriate modelLLM or SLMfor a given task,routes inputs accordingly,and combines their outputs into

110、 a unified response.Privacy ConcernsPrivacy Concerns4040 The growing adoption of AI agents powered by LLMs,brings the promise of revolutionizing the way humans work by automating tasks and improving productivity.However,these systems also introduce significant privacy risks that need to be carefully

111、 managed:36 Cabalar,R.,What are small language models?(2024)https:/ 37 D.Biswas,ICAART,Stateful Monitoring and Responsible Deployment of AI Agents,(2025)38 Windland,V.et al.What is LLM orchestration(2024)https:/ 39 D.Vellante et al.,From LLMs to SLMs to SAMs,how agents are redefining AI(2024)https:/

112、 40 B.ONeill,What is an AI agent?A computer scientist explains the next wave of artificial intelligence tools(2024)https:/ AI Privacy Risks&Mitigations Large Language Models(LLMs)15 To perform their tasks effectively,AI agents often require access to a wide range of user data,such as:o Internet acti

113、vity:Browsing history,online searches,and frequently visited websites.o Personal applications:Emails,calendars,and messaging apps for scheduling or communication tasks.o Third-party systems:Financial accounts,customer management platforms,or other organizational systems.This level of access signific

114、antly increases the risk of unauthorized data exposure,particularly if the agents systems are compromised.AI agents are designed to make decisions autonomously,which can lead to errors or choices that users may disagree with.Like other AI systems,AI agents are susceptible to biases originating from

115、their training data,algorithms and usage context.Privacy trade-offs for user convenience:41 As AI agents grow more capable,users will need to consider how much personal data they are willing to share in exchange for convenience.For example,an agent might save time by managing travel bookings or nego

116、tiating purchases but requires access to sensitive information such as payment details or login credentials42.Balancing these trade-offs requires clear communication about data usage policies and robust consent mechanisms.Accountability for Agent decisions:43 AI agents operate in complex environment

117、s and may encounter unforeseen challenges.When an agent makes an error,or its actions cause harm,determining accountability can be difficult.Organizations must ensure transparency in how decisions are made and provide mechanisms for users to intervene when errors occur.Common Uses of LLM Systems Com

118、mon Uses of LLM Systems LLMs have become pivotal in various industries,offering advanced capabilities in natural language understanding and generation.The market provides a spectrum of LLM solutions,each tailored to specific applications and user requirements.1.Proprietary LLM Models Leading technol

119、ogy companies have developed proprietary LLM platforms that cater to diverse business needs.Some platforms offer customizable LLMs that can be trained on specific datasets:OpenAIs GPT44 Series(Generative Pre-Trained Transformer(GPT)models),are renowned for their advanced language processing capabili

120、ties.These models are accessible through APIs,enabling businesses to integrate sophisticated language understanding and generation into their applications.Googles Gemini45 models are designed to assist with various tasks,providing users with detailed information and facilitating complex queries.Clau

121、des Anthropic Models46 are developed with safety and alignment in mind.Claude specializes in conversational AI with a focus on ethical and secure interactions.Several European companies and collaborations are contributing to the LLM landscape:41 Z.Zhang et al.Its a Fair Game,or Is It?Examining How U

122、sers Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents(2024)https:/arxiv.org/abs/2309.11653 42 Login credentials are the unique information used to access systems,accounts,or services,typically consisting of a username and password,but they can also include additional

123、 methods like two-factor authentication,biometric data,or security PINs for added protection.43 J.Zeiser,Owning Decisions:AI Decision-Support and the Attributability-Gap(2024).https:/doi.org/10.1007/s11948-024-00485-1 44 ChatGPT(https:/ Gemini(https:/ Claude(https:/claude.ai/)AI Privacy Risks&Mitiga

124、tions Large Language Models(LLMs)16 Mistral AI,47a Paris-based startup established in 2023 by former Google DeepMind and Meta AI scientists offers both open source and proprietary AI models.Aleph Alpha48is based in Heidelberg,Germany,and it specializes in developing LLMs designed to provide transpar

125、ency regarding the sources used for generating results.Their models are intended for use by enterprises and governmental agencies,trained in multiple European languages.Silo AIs Poro49,through its generative AI arm SiloGen,has developed Poro,a family of multilingual open source LLMs.This initiative

126、aims to strengthen European digital sovereignty and democratize access to LLMs for all European languages.TrustLLM50is a coordinated project by Linkping University that focuses on developing trustworthy and factual LLM technology for Europe,emphasizing accessibility and reliability.OpenEuroLLM51 is

127、an open source family of performant,multilingual,large language foundation models for commercial,industrial and public services.2.Open Source LLM Frameworks and Models The open source community contributes significantly to the LLM landscape.Here are some of the most known frameworks and models that

128、have shaped the development and deployment of large language models:Hugging Faces Transformers52 is an extensive library of pre-trained models and tools,allowing developers to fine-tune and deploy LLMs for specific tasks.Deepseek53 is an advanced language model comprising 67 billion parameters.It ha

129、s been trained from scratch on a vast dataset of 2 trillion tokens in both English and Chinese.Deepsets Haystack54 is an open source framework designed to build search systems and question-answering applications powered by Large Language Models(LLMs)and other natural language processing(NLP)techniqu

130、es.OLMo 32B55 is the first fully open model(all data,code,weights,and details are freely available).Metas LLaMA56 models focus on research and practical applications in NLP.BLOOM57 was developed by BigScience as a multilingual open source model capable of generating text in over 50 languages,with a

131、focus on accessibility and inclusivity.BERT58 was created by Google to understand the context of text through bidirectional language representation,excelling in tasks like question answering and sentiment analysis.Falcon59 was developed by the Technology Innovation Institute as a high-performance mo

132、del optimized for text generation and understanding,with significant efficiency improvements over similar models.Qwen60 is a large language model family built by Alibaba Cloud.47 Mistral(https:/mistral.ai/)48 Aleph Alpha(https:/aleph- Silo AI,Poro-a family of open models that bring European language

133、s to the frontier(2023)https:/www.silo.ai/blog/poro-a-family-of-open-models-that-bring-european-languages-to-the-frontier 50 TrustLLM(https:/trustllm.eu/)51 OpenEuroLLM(https:/openeurollm.eu/)52 Hugging Face,Transformers(n.d)https:/huggingface.co/docs/transformers/v4.17.0/en/index 53 Deepseek(https:

134、/ Haystack(https:/haystack.deepset.ai/)55 Ai2,OLMo 2 32B:First fully open model to outperform GPT 3.5 and GPT 4o mini(2025)https:/allenai.org/blog/olmo2-32B 56 Llama(https:/ Hugging Face,Introducing The Worlds Largest Open Multilingual Language Model:BLOOM(2025)https:/bigscience.huggingface.co/blog/

135、bloom 58 Hugging Face,BERT(n.d)https:/huggingface.co/docs/transformers/model_doc/bert 59 TTI,Introducing the Technology Innovation Institutes Falcon 3(n.d)https:/falconllm.tii.ae/60 Hugging Face,Qwen(n.d)https:/huggingface.co/Qwen AI Privacy Risks&Mitigations Large Language Models(LLMs)17 LangChain6

136、1 is an open source framework for building applications powered by large language models.3.Cloud-Based LLM Services Major cloud providers offer LLM services that integrate seamlessly into existing infrastructures providing access to proprietary and open source LLMs:Microsoft Azure OpenAI62 Service c

137、ollaborates with OpenAI to provide API access to GPT models,enabling businesses to incorporate advanced language features into their applications.Amazon Web Services(AWS)Bedrock63 offers a suite of AI services,including language models that support various natural language processing tasks.Google Cl

138、oud Vertex AI64 is a platform for building,deploying,and scaling machine learning models,including LLMs.It provides access to models like PaLM 2 and supports customization for various applications,such as translation,summarization,and conversational AI.IBM Watson65 provides LLM capabilities that can

139、 be tailored to recognize industry-specific entities,enhancing the relevance and accuracy of information extraction.Cohere66 offers customizable LLMs that can be fine-tuned for specific tasks.Applications of LLMs Applications of LLMs LLMs are employed across various applications67,enhancing both use

140、r experience and operational efficiency.This list represents some of the most prominent applications of LLMs,but it is by no means exhaustive.The versatility of LLMs continues to unlock new use cases across industries,demonstrating their transformative potential in various domains.Chatbots and AI As

141、sistants:68 LLMs power virtual assistants like Siri,Alexa,and Google Assistant,understand and process natural language,interpret user intent,and generate responses.Content generation:69 LLMs assist in creating articles,reports,and marketing materials by generating human-like text,thereby streamlinin

142、g content creation processes.Language translation:70 Advanced LLMs facilitate real-time translation services.Sentiment analysis:71 Businesses use LLMs to analyze customer feedback and social media content,gaining insights into public sentiment and informing strategic decisions.Code generation and de

143、bugging:72 Developers leverage LLMs to generate code snippets and identify errors,enhancing software development efficiency.Educational support tools:73 LLMs play a key role in personalized learning by generating educational content,explanations,and answering student questions.Legal document process

144、ing:74 LLMs help professionals in the legal field by reviewing and summarizing legal texts,extracting important information,and offering insights.61 LangChain,Introduction(n.d)https:/ Microsoft,Azure OpenAI Service(2025)https:/ AWS,Bedrock(n.d)https:/ 64 Vertex AI Platform,Innovate faster with enter

145、prise-ready AI,enhanced by Gemini models(n.d)https:/ 65 IBM,IBM Watson to watsonx(n.d)https:/ 66 Cohere(https:/ N.Sashidharan,Three Pillars of LLM:Architecture,Use Cases,and Examples (2024)https:/ 68 Google Assistant(https:/ Jasper AI(https:/www.jasper.ai/)70 Deepl(https:/ SurveySparrow(https:/ GitH

146、ub Copilot(https:/ Khanmigo(https:/www.khanmigo.ai/)74 Luminance(https:/ Privacy Risks&Mitigations Large Language Models(LLMs)18 Customer support:75 Automating responses to customer inquiries and escalating complex cases to human agents.Autonomous vehicles:76 Driving cars with real-time decision-mak

147、ing capabilities.Performance Measures for LLMs Performance Measures for LLMs Evaluating the performance of Large Language Models(LLMs)is essential to ensure they meet their intended purpose and desired standards of accuracy,reliability,and ethical use across diverse applications.To effectively measu

148、re the performance of a Large Language Model(LLM),it is important to tailor the evaluation approach to the stage of the LLM lifecycle(e.g.,training,post-processing,pre-deployment,production)and its intended real-world applications.Performance metrics help identify areas where additional testing or r

149、efinements may be necessary before deployment or once the LLM system is in use in a production environment.Some of the most common LLM performance evaluation criteria are Answer Relevancy,Correctness,Semantic Similarity,Fluency,Hallucination,Factual Consistency,Contextual Relevancy,Toxicity,Bias and

150、 Task-Specific Metrics.The following metrics77 are commonly used,each offering different insights:Accuracy78measures how often an output aligns with the correct or expected results.In tasks like text classification or question answering,accuracy is calculated as the ratio of correct predictions to t

151、he total number of predictions.However,for generative tasks such as text generation,traditional accuracy metrics may not fully capture performance due to the open-ended nature of possible correct responses.In such cases,metrics like BLEU(Bilingual Evaluation Understudy)and ROUGE(Recall-Oriented Unde

152、rstudy for Gisting Evaluation)are employed to assess the quality of generated text by comparing it to reference texts.Precision quantifies the ratio of correctly predicted positive outcomes to the total number of positive predictions made by the model.In the context of LLMs,a high precision score in

153、dicates the model is accurate when making predictions.However,it does not account for relevant instances the model fails to predict(false negatives),so it is commonly combined with recall for a more comprehensive evaluation.Recall,also referred to as sensitivity or the true positive rate,measures th

154、e proportion of actual positive instances that the model successfully identifies.A high recall score reflects the models effectiveness in capturing relevant information but does not address irrelevant predictions(false positives).For this reason,recall is typically evaluated alongside precision to p

155、rovide a balanced view.F1 Score offers a balanced metric by combining precision and recall into their harmonic mean.A high F1 score indicates that the model achieves a strong balance between precision and recall,making it a valuable metric when both false positives and false negatives are critical.T

156、he F1 score ranges from 0 to 1,with 1 representing perfect performance on both metrics.Specificity79 measures the proportion of true negatives correctly identified by a model.AUC(Area Under the Curve)and AUROC80(Area Under the Receiver Operating Characteristic Curve)quantify a models ability to dist

157、inguish between classes.It evaluates the trade-off between 75 Salesforce(https:/ Tesla Autopilot(https:/ A.Chaudhary,Understanding LLM Evaluation and Benchmarks:A Complete Guide(2024)https:/ 78 S.Karzhev,LLM Evaluation:Metrics,Methodologies,Best Practices(2024)https:/ 79 Wikipedia,Sensitivity and Sp

158、ecificity(2025)https:/en.wikipedia.org/wiki/Sensitivity_and_specificity 80 E.Becker and S.Soatto,Cycles of Thought:Measuring LLM Confidence through Stable Explanations(2024)https:/arxiv.org/pdf/2406.03441v1 AI Privacy Risks&Mitigations Large Language Models(LLMs)19 sensitivity(true positive rate)and

159、 1-specificity(false positive rate)across various thresholds.A higher AUC value indicates better performance in classification tasks.AUPRC81(Area Under the Precision-Recall Curve),measures a models performance in imbalanced datasets,focusing on the trade-off between precision and recall.A high AUPRC

160、 indicates that the model performs well in identifying positive instances,even when they are rare.Cross Entropy82is a measure of uncertainty or randomness in a systems predictions.It measures the difference between two probability distributions:the true labels(actual data distribution)and the predic

161、ted probabilities from the model(output).Lower entropy means higher confidence in predictions,while higher entropy indicates uncertainty.Perplexity83derives from cross entropy and evaluates how well a language model predicts a sample,serving as an indicator of its ability to handle uncertainty.A low

162、er perplexity score means better performance,indicating that the model is more confident in its predictions.Some studies suggest that perplexity has proven unreliable84 to evaluate LLMs due to their long-context capabilities.It is also difficult to use perplexity as a benchmark between models since

163、its scores depend on factors like tokenization method,dataset,preprocessing steps,vocabulary size,and context length.85 Calibration86refers to the alignment between a models predicted probabilities and the actual probability of those predictions being correct.A well-calibrated model provides confide

164、nce scores that accurately reflect the true probabilities of outcomes.Proper calibration is vital in applications where understanding the certainty of predictions is important,such as in medical diagnoses or legal document analysis.MoverScore87is a modern metric developed to assess the semantic simi

165、larity between two texts.Other metrics used for assessing the performance and usability of LLM-based systems,especially in real-time or high-demand applications are:88 Completed requests per minute:Measures how many requests the LLM can process and return responses for in one minute.It reflects the

166、systems efficiency in handling multiple queries.Time to first token(TTFT):The time taken from when a request is submitted to when the first token of the response is generated.Inter-token Latency(ITL):The time delay between generating consecutive tokens in the response.This metric evaluates the speed

167、 and fluidity of text generation.End to end Latency/ETEL):The total time taken from when a request is made to when the entire response is completed.It encompasses all processing stages,including input handling,model inference,and output generation.81 J.Czakon,F1 Score vs ROC AUC vs Accuracy vs PR AU

168、C:Which Evaluation Metric Should You Choose?(2024)https:/neptune.ai/blog/f1-score-accuracy-roc-auc-pr-auc 82 C.Xu,Understanding the Role of Cross-Entropy Loss in Fairly Evaluating Large Language Model-based Recommendation(2024)https:/arxiv.org/pdf/2402.06216v2 83 C.Huyen Evaluation Metrics for Langu

169、age Modeling(2019)https:/thegradient.pub/understanding-evaluation-metrics-for-language-models/84 L.Fang What is wrong with perplexity for long-context language modeling?(2024)https:/arxiv.org/pdf/2410.23771v1 85 A.Morgan Perplexity for LLM Evaluation(2024)https:/ P.Liang et al.Holistic Evaluation of

170、 Language Models(2023)https:/arxiv.org/abs/2211.09110 87 PI,Moverscore 1.0.3(2020)https:/pypi.org/project/moverscore/88 W.Kadous et al.Reproducible Performance Metrics for LLM inference(2023)https:/ AI Privacy Risks&Mitigations Large Language Models(LLMs)20 In addition to these metrics,there are com

171、prehensive evaluation frameworks or benchmarks89 such as GLUE(General Language Understanding Evaluation)90,MMLU(Massive Multitask Language Understanding)91,HELM(Holistic Evaluation of Language Models)92,DeepEval93 or OpenAI Evals94.Task-specific metrics such as BLEU95(Bilingual Evaluation Understudy

172、),ROUGE96(Recall-Oriented Understudy for Gisting Evaluation),and BLEURT97(Bilingual Evaluation Understudy with Representations from Transformers)are widely used for evaluating text generation,summarization,and translation.It is important to recognize that quantitative metrics alone are not sufficien

173、t.While these metrics are highly valuable in identifying risks,especially when integrated into automated evaluation pipelines,they primarily serve as early warning signals,prompting further investigation when thresholds are exceeded.Many critical risks,including misuse potential,ethical concerns,and

174、 long-term impact,cannot be effectively captured through those numerical measurements alone.To ensure a more holistic evaluation,organizations should complement quantitative indicators with expert judgment,scenario-based testing,and qualitative assessments.Open source frameworks like Inspect98,suppo

175、rt an integrated approach by enabling model-graded evaluations,prompt engineering,session tracking,and extensible scoring techniques.These tools help operationalize both metric-based and qualitative evaluations,offering better observability and insight into LLM behavior in real-world settings.Measur

176、ing Performance in Agentic AI Measuring Performance in Agentic AI Most current metrics99 for AI agents focus on efficiency,effectiveness,and reliability.These include system metrics(resource consumption and technical performance),task completion(measuring goal achievement),quality control(ensuring o

177、utput consistency),and tool interaction(evaluating integration with external tools and APIs).Some of the key metrics used include:Task-specific accuracy:100 Assesses how correctly the agent performs designated tasks,such as classification or information retrieval.Metrics like Exact Match(EM)and F1 S

178、core are commonly used.End-to-end task completion:101 Evaluates the agents ability to achieve user-defined goals through a series of actions.Metrics include Task Success Rate(TSR)and Goal Completion Rate(GCR).Step-Level accuracy:Assesses the correctness of individual actions taken by the agent withi

179、n a larger workflow.This is critical in multi-step processes,such as booking a service or resolving a technical issue.89 Benchmarks are standardized frameworks developed to assess LLMs across various scenarios and metrics(See also section 10 of this document).90 Gluebenchmark(https:/ Papers with cod

180、e,MMLU(Massive Multitask Language Understanding)(n.d)https:/ 92 Center for Research on Foundation Models,A reproducible and transparent framework for evaluating foundation models(n.d)https:/crfm.stanford.edu/helm/93 GitHub,The LLM Evaluation framework(n.d)https:/ 94 GitHub,Evals is a framework for e

181、valuating LLMs and LLM systems,and an open-source registry of benchmarks(n.d)https:/ 95 Wikipedia,BLEU(2025)https:/en.wikipedia.org/wiki/BLEU 96 Wikipedia,ROUGE(metric)(2025)https:/en.wikipedia.org/wiki/ROUGE_(metric)97 GitHub,BLEURT is a metric for Natural Language Generation based on transfer lear

182、ning(n.d)https:/ 98 AISI,An open-source framework for large language model evaluations(n.d)https:/inspect.aisi.org.uk/99 P.Bhavsar Mastering Agents:Metrics for Evaluating AI Agents(2024)https:/www.galileo.ai/blog/metrics-for-evaluating-ai-agents 100 https:/ AISERA,An Introduction to Agent Evaluation

183、(n.d)https:/ Privacy Risks&Mitigations Large Language Models(LLMs)21 Precision and Recall:Measures how accurately the agent retrieves relevant information(precision)and whether it captures all necessary details(recall).These metrics are vital for tasks like document summarization or answering comple

184、x queries.Contextual understanding:102Measures the agents proficiency in maintaining and utilizing context in interactions,crucial for coherent multi-turn dialogues.Dialog State Tracking103 is a relevant metric.User satisfaction:104Measures user perceptions of the agents performance,often through fe

185、edback scores or surveys and using scales to measure system and user experience usability.Evaluating AI agents with traditional LLM benchmarks presents challenges,as they often fail to capture real-world dynamics,multi-step reasoning,tool use,and adaptability.Effective assessment requires new benchm

186、arks that measure long-term planning,interaction with external tools,and real-time decision-making.Below are some of the most recognized benchmarks currently used:SWE-bench:105Software Engineering Benchmark dataset,created to systematically evaluate the capabilities of an LLM in resolving software i

187、ssues.AgentBench:106107 It is designed for evaluating and training visual foundation agents based on LMMs.MLAgentBench:108 To evaluate if agents driven by LLMs perform machine learning experimentation effectively.BFCL(Berkeley Function-Calling Leaderboard):109 To evaluate the ability of different LL

188、Ms to call functions(also referred to as tools).-bench:110 A benchmark for tool-agent-user interaction in real-world domains.Planbench:111 To evaluate LLMs on planning and reasoning.Issues that can Affect the Accuracy of the Output Issues that can Affect the Accuracy of the Output Several factors ca

189、n impact the accuracy of the outputs generated by LLMs.Understanding these issues is essential for optimizing their performance and mitigating risks in practical applications.Some of the more common issues are:1.Quality of training data Data bias:112If the training data contains biases(e.g.,societal

190、,cultural,or linguistic biases),the model may replicate or amplify these biases in its outputs.Data relevance:113Training on outdated,irrelevant,or noisy data can lead to inaccurate or contextually irrelevant responses.102 Smyth OS,Conversational Agents and Context Awareness:How AI Understands and A

191、dapts to User Needs(n.d)https:/ Papers with code,Dialogue State Tracking,(n.d)https:/ 104 N.Bekmanis,Artificial Intelligence Conversational Agents:A Measure of Satisfaction in Use(2023)https:/essay.utwente.nl/94906/1/Bekmanis_MA_BMS.pdf 105 Swebench(https:/ Github,A Comprehensive Benchmark to Evalua

192、te LLMs as Agents(ICLR24),(n.d)https:/ 107 Papers with code,Agentench(n.d)https:/ 108 Q.Huang et al.MLAgentBench:Evaluating Language Agents on Machine Learning Experimentation(2024)https:/arxiv.org/abs/2310.03302 109 Hugging Face Dataset(https:/huggingface.co/datasets/gorilla-llm/Berkeley-Function-C

193、alling-Leaderboard)110 GitHub,Code and Data(n.d)https:/ 111 GitHub,An extensible benchmark for evaluating large language models on planning(n.d)https:/ I.O.Gallegos et al.Bias and Fairness in Large Language Models:A Survey(2024)https:/direct.mit.edu/coli/article/50/3/1097/121961/Bias-and-Fairness-in

194、-Large-Language-Models-A 113 Large Language Models pose risk to science with false answers,says Oxford study(2023)https:/www.ox.ac.uk/news/2023-11-20-large-language-models-pose-risk-science-false-answers-says-oxford-study AI Privacy Risks&Mitigations Large Language Models(LLMs)22 2.Model limitations

195、 Understanding context:114Despite advanced architectures,LLMs can struggle with nuanced contexts or multi-turn conversations where earlier parts of the dialogue must inform later responses.Handling ambiguities:115Ambiguous input can lead to incorrect or nonsensical outputs if the model cannot infer

196、the intended meaning.3.Tokenization and preprocessing Tokenization errors:116Misrepresentation of input text due to tokenization issues(e.g.,splitting words incorrectly)can distort model understanding.Preprocessing issues:117Overly aggressive cleaning or normalization during preprocessing can remove

197、 important contextual information,reducing accuracy.4.Overfitting and underfitting Overfitting:118Training for too many iterations on a limited dataset can make the model overly specialized,leading to poor performance on unseen data.Underfitting:119Inadequate training or overly simple models may fai

198、l to capture the complexity of the task,resulting in general inaccuracies.5.Prompt design and input quality Prompt sensitivity:120LLMs are highly sensitive to how inputs are phrased.Minor variations in prompt structure can lead to drastically different outputs.Garbage in,garbage out:121Poorly worded

199、 or unclear input can lead to inaccurate or irrelevant responses.6.Limitations in knowledge Knowledge cutoff:122LLMs are trained on data up to a specific point in time.They may lack awareness of recent developments or emerging knowledge.Factual errors:123LLMs can hallucinate information,generating p

200、lausible but factually incorrect responses due to the probabilistic nature of their predictions.7.Lack of robustness Adversarial inputs:124LLMs may fail when presented with deliberately manipulated or adversarial inputs designed to exploit their weaknesses.114 J.Browning,Getting it right:the limits

201、of fine-tuning large language models(2024)https:/ 115 E.Jones and J.Steinhardt,Capturing Failures of Large Language Models via Human Cognitive Biases(2022)https:/arxiv.org/abs/2202.12299 116 G.B.Mohan et al.An analysis of large language models:their impact and potential application(2024)https:/ 117

202、H.Naveed et al A Comprehensive Overview of Large Language Models(2024)https:/arxiv.org/abs/2307.06435 118 P.Jindal Evaluating Large Language Models:A Comprehensive Guide(2024)https:/ 119 idem 120 J.Browning Getting it right:the limits of fine-tuning large language models(2024)https:/ 121 H.Naveed et

203、 al.A Comprehensive Overview of Large Language Models(2024)https:/arxiv.org/abs/2307.06435 122 University of Oxford,Large Language Models pose risk to science with false answers,says Oxford study(2023)https:/www.ox.ac.uk/news/2023-11-20-large-language-models-pose-risk-science-false-answers-says-oxfo

204、rd-study 123 ht Ho,D.E.,Hallucinating Law:Legal Mistakes with Large Language Models are Pervasive(2024)https:/hai.stanford.edu/news/hallucinating-law-legal-mistakes-large-language-models-are-pervasive 124 E.Jones and J.Steinhardt,Capturing Failures of Large Language Models via Human Cognitive Biases

205、(2022)https:/arxiv.org/abs/2202.12299 AI Privacy Risks&Mitigations Large Language Models(LLMs)23 Noise and variability:125Spelling errors,slang,or non-standard language can lead to misinterpretations and lower accuracy.8.Inadequate calibration Overconfidence:126Poorly calibrated models may assign hi

206、gh confidence scores to incorrect predictions,misleading users.Failing to properly convey uncertainty in predictions can erode trust in the model.125 G.B.Mohan An analysis of large language models:their impact and potential applications(2024)https:/ 126 L.Li et al.Confidence Matters:Revisiting Intri

207、nsic Self-Correction Capabilities of Large Language Models(2024)https:/arxiv.org/abs/2402.12563 AI Privacy Risks&Mitigations Large Language Models(LLMs)24 3.3.Data Flow and Associated Privacy Risks in LLM Systems Data Flow and Associated Privacy Risks in LLM Systems Understanding the data flow in AI

208、 systems powered by LLMs is crucial for assessing privacy risks.This flow may vary depending on the phases of operation,the specific system the model integrates,and the type of service model in use,each of which introduces unique challenges for data protection.The Importance of the AI Lifecycle in P

209、rivacy Risk Management The Importance of the AI Lifecycle in Privacy Risk Management The lifecycle of an AI system,as outlined in standards ISO/IEC 22989127and ISO/IEC 5338,128provides a structured framework for understanding the flow of data throughout the development,deployment,and operation of AI

210、 systems.This lifecycle is also essential for identifying and mitigating privacy risks at each stage.Figure 7.Source:Based on ISO/IEC 22989 In this document,we use this AI lifecycle as a reference framework,recognizing that each organization may have its own adapted version based on its specific nee

211、ds.While the core stages of the lifecycle are generally similar across organizations,the exact phases may vary.Each one of the phases of the lifecycle involves unique privacy risks that require tailored mitigation strategies.Implementing Privacy by Design into each phase helps to address risks proac

212、tively rather than retroactively fixing them.127 ISO/IEC 22989(Artificial Intelligence Concepts and Terminology)128 ISO/IEC 5338:2023 Information technology Artificial intelligence AI system life cycle processes AI Privacy Risks&Mitigations Large Language Models(LLMs)25 AI Lifecycle Phases and their

213、 Impact on Privacy AI Lifecycle Phases and their Impact on Privacy 1.Inception and Design:In this phase,decisions are made regarding data requirements,collection methods,and processing strategies.The selection of data sources may introduce risks if sensitive or personal data is included without adeq

214、uate safeguards.2.Data Preparation and Preprocessing:Raw data is collected,cleaned,in some cases anonymized129,and prepared for training or fine-tuning.Datasets are often sourced from diverse origins,including web-crawled data,public repositories,proprietary data,or datasets obtained through partner

215、ships and collaborations.Privacy risks:o Training data may inadvertently include personal details,confidential documents,or other sensitive information.o Inadequate anonymization or handling of identifiable data can lead to breaches or unintended inferences during later stages.o Biases present in th

216、e datasets can affect the models predictions,resulting in unfair or discriminatory outcomes.o Errors or gaps in training data can adversely impact the models performance,reducing its effectiveness and reliability.o The collection and use of training data may violate privacy rights,lack proper consen

217、t,or infringe on copyrights and other legal obligations.3.Development,Model Training:Prepared datasets are used to train the model,which involves large-scale processing.The model may inadvertently memorize sensitive data,leading to potential privacy violations if such data is exposed in outputs.4.Ve

218、rification&Validation:130 The model is evaluated using test datasets,often including real-world scenarios.Testing data may inadvertently expose sensitive user information,particularly if real-world datasets are used without anonymization.5.Deployment:The model interacts with live data inputs from us

219、ers,often in real-time applications that could integrate with other systems.Live data streams might include highly sensitive information,requiring strict controls on collection,transmission,and storage.6.Operation and Monitoring:Continuous data flows into the system for monitoring,feedback,and perfo

220、rmance optimization.Logs from monitoring systems may retain personal data such as user interactions,creating risks of data leaks or misuse.7.Re-evaluation,Maintenance and Updates:Additional data may be collected for retraining or updating the model to improve accuracy or address new requirements.Usi

221、ng live user data for updates without proper consent or safeguards can violate privacy principles.8.Retirement:Data associated with the model and its operations is archived or deleted.Failure to properly erase personal data during decommissioning can lead to long-term privacy vulnerabilities.Through

222、out the AI system lifecycle,it is important to consider how different types of personal data may be involved at each phase.Depending on the stage,personal data can be collected,processed,exposed,or transformed in different ways.Recognizing this variability is essential for implementing effective pri

223、vacy and data protection measures.129 Important to consider the EDPB opinion 28/2024 and section 3.2 On the circumstances under which AI models could be considered anonymous and the related demonstration:,the EDPB considers that,for an AI model to be considered anonymous,using reasonable means,both(

224、i)the likelihood of direct(including probabilistic)extraction of personal data regarding individuals whose personal data were used to train the model;as well as(ii)the likelihood of obtaining,intentionally or not,such personal data from queries,should be insignificant for any data subject.130 Testin

225、g,Evaluation,Validation,and Verification(TEVV)is an ongoing process that occurs throughout the AI lifecycle to ensure that a system meets its intended requirements,performs reliably,and aligns with safety and compliance standards.AI Privacy Risks&Mitigations Large Language Models(LLMs)26 Data Flow a

226、nd Privacy Risks per LLM Service Model Data Flow and Privacy Risks per LLM Service Model It is common to encounter terms like closed models,open models,closed weights,and open weights in the context of LLMs.Understanding these terms is essential for assessing the risks associated with different mode

227、l release strategies.Closed models are proprietary models that do not provide public access to their weights or source code and interaction with the model is restricted,typically requiring an API or subscription,while open models are made publicly available fully(weights,full code,training data,and

228、other documentation is available)or partly(not everything is available,usually training data;or it is available under licences).Similarly,closed weights indicate proprietary models whose trained parameters are not disclosed,whereas open weights describe models with publicly available parameters,allo

229、wing for inspection,fine-tuning,or integration into other systems.It is also important to distinguish the term open model from open source model.This classification of a model as open source requires it to be released under an open source license,which legally grants anyone the freedom to use,study,

230、modify,and distribute the model for any purpose131.Term Privacy Risks Closed models&closed weights Often minimal external transparency.Users rely entirely on the providers privacy safeguards,making it difficult to independently verify compliance with data protection regulations.Open models&open weig

231、hts Risk of personal data exposure and security breaches if training data contains sensitive or harmful content.Partial access may prevent full scrutiny of model training data and privacy vulnerabilities.Open source Open source models share the same privacy risks as open models and open weight model

232、s.While open source fosters transparency and innovation,it also increases risks,as modifications may introduce security vulnerabilities or remove built-in safety measures.LLMs are predominantly accessible through the following service models:131 AI Action Summit,International AI Safety Report on the

233、 Safety of Advanced AI,p 150,(2025)https:/assets.publishing.service.gov.uk/media/679a0c48a77d250007d313ee/International_AI_Safety_Report_2025_accessible_f.pdf Figure 8.The illustration shows how different types of personal data can arise across various phases of the AI lifecycle.AI Privacy Risks&Mit

234、igations Large Language Models(LLMs)27 1.LLM as a Service:This service model provides access to LLMs via APIs hosted on a cloud platform.Users can send input and receive output without having direct access to the models underlying architecture or weights.Based on this service model we can usually fi

235、nd the different LLM model variations available:o Closed models with closed weights where the provider trains the model and retains control over the weights and data,offering access through an API.This approach ensures ease of use but requires user data to flow through the providers systems.Example:

236、OpenAI GPT-4 API132 o Customizable closed weights where deployers may fine-tune the model using their own data,within a controlled environment,although the underlying weights remain inaccessible balancing customization with security.Example:Azure OpenAI Service133 o Open weights where some providers

237、 grant deployers full or partial access to the architecture for greater transparency and flexibility through a platform or via an API134.Example:Hugging Faces models in AWS Bedrock135 2.LLM off-the-shelf:In this service model the deployer can customize weights and fine tune the model.This happens so

238、metimes through platforms like Microsoft Azure and AWS where a deployer can select a model and develop their own solution with it.It is also commonly used with open weight models,such as LLaMA or BLOOM.While an LLM as a Service typically involves API-based interaction without model ownership,the LLM

239、 off-the-Shelf service emphasizes more developer and deployer control.The distinction lies in this level of control and access provided,for instance,in Hugging Face models can be downloaded locally.3.Self-developed LLM:In this model,organizations develop and deploy LLMs on their own infrastructure,m

240、aintaining full control over data and model interaction.While this option may offer more privacy,this service model requires of significant computational resources and expertise.Each of the three service models features a distinct data flow.While there are similarities across models,each phasefrom u

241、ser input to output generationpresents unique risks that can impact user privacy and data protection.In this section,we will first examine the data flow in an LLM as a Service solution,followed by an analysis of the key differences in data flow when using an LLM off-the-shelf model and a self-develo

242、ped LLM system.*Note that in this section,the terms provider136 and deployer137 are used as defined in the AI Act,where the provider refers to the entity developing and offering the AI system,and the deployer refers to the entity implementing and operating the system for end-users.132 Open AI,The mo

243、st powerful platform for building AI products,(2025)https:/ Microsoft,Azure OpenAI Service(2025)https:/ Wikipedia,API(2025)https:/en.wikipedia.org/wiki/API 135 S.Pagezy,Use Hugging Face models with Amazon Bedrock(2024)https:/huggingface.co/blog/bedrock-marketplace 136 provider means a natural or leg

244、al person,public authority,agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark,whether for payment or free of char

245、ge;(Article 3(3)AI Act)137 deployer means a natural or legal person,public authority,agency or other body using an AI system under its authority except where the AI system is used in the course of a personal non-professional activity;(Article 3(4)AI Act)AI Privacy Risks&Mitigations Large Language Mo

246、dels(LLMs)28 1.Data Flow in a LLM as a Service System 1.Data Flow in a LLM as a Service System In our example,a user interacts with an LLM application hosted online by a provider.This data flow focuses solely on the phases involved during the users interaction with the service.Model preparation,depl

247、oyment,and integration by the provider are outside the scope since they will be examined further in the self-developed LLM system example.It is important to note that each use case will have its own specific data flow depending on its unique requirements and context and the examples provided in this

248、 section are intended to be generic representations.In an LLM as a Service scenario we could find these general data flow phases:User input:The process starts with the user submitting input,such as a query or command.This could be entered through a web-based interface,mobile application,or other too

249、ls provided by the LLM provider.Provider interface&API:The input is sent through an interface or application managed by the provider(e.g.,a webpage,app or a chatbot window embedded on a website).This interface ensures the input is formatted appropriately and securely transmitted to the LLM infrastru

250、cture.LLM processing at providers infrastructure:The API receives the input and routes it to the LLM model hosted on the providers infrastructure.The LLM processes the input using its trained parameters(weights)to generate a relevant response.This may involve steps like tokenization,context understa

251、nding,reasoning,and text generation.The model generates a response.*Logging:The provider may log the user input(query)along with the generated response to analyze the interaction and identify system errors or gaps in response quality.The data could be also included in a training dataset to improve t

252、he models ability to handle similar queries in the future.In this case,anonymization and filtering techniques are often applied.Processed output:The generated output is returned via the providers interface to the user.The response is typically in a format ready for display or integration,such as tex

253、t,suggestions,or actionable data.Privacy considerations in this data flow The following table highlights potential privacy and data protection risks and their recommended mitigations.Figure 9.Data Flow in a LLM as a Service System AI Privacy Risks&Mitigations Large Language Models(LLMs)29 Phases Pos

254、sible Risks&Mitigations User input Risks:Sensitive data disclosure:Users may unknowingly or inadvertently input sensitive personal data,such as names,addresses,financial information,or medical details.Unauthorized access:If the web interface,application,databases or input tool lacks robust access co

255、ntrols,unauthorized individuals138 may gain access to user accounts or systems,allowing them to view previously submitted data or queries.Lack of transparency:Users may not be fully aware of how their data will be used,retained,or shared by the provider.Adversarial attacks:A(malicious)user might cra

256、ft input designed to manipulate the LLMs behavior or bypass its intended functionality,such as injecting unauthorized instructions into queries(prompt injection attack),or users may try to bypass safety restrictions139 imposed on the model by crafting specific input(jailbreaking attempt)140.Mitigati

257、ons for Providers:-Implement clear user guidance and input restrictions,such as filters or warnings to discourage the entry of personal data.Use automated detection mechanisms141142 to flag or anonymize sensitive information before it is processed or logged.-Encrypt user inputs and outputs during bo

258、th transmission and storage to protect sensitive data from unauthorized access.Insufficient encryption can expose user queries and data to potential breaches,especially during the user input phase.Ensure encryption in transit using robust protocols(e.g.,TLS)and encryption at rest for stored data.Add

259、itionally,implement data segregation practices to isolate user data,preventing unauthorized individuals from accessing or compromising multiple accounts or datasets.Another mitigation to prevent unauthorized access is the implementation of secure password practices based on the latest NIST143 and EN

260、ISA144 recommendations:require a minimum password length of 8 characters,update passwords if they are compromised or forgotten,enforce the use of multifactor authentication(MFA),ensure passwords differ significantly from previous ones,check passwords against blacklists,enforce account lockout polici

261、es,monitor failed login attempts,discourage the use of password hints,and store passwords securely using hashing and salting techniques with robust algorithms such as bcrypt,Argon2,or PBKDF2.-Inform users about how their data will be used,retained,and processed through clear and easily accessible pr

262、ivacy policies.-Though there are currently no foolproof measures145 to protect against prompt injection and jailbreaking,146147some of the most common best practices include the 138 G.Nagli Wiz Research Uncovers Exposed DeepSeek Database Leaking Sensitive Information,Including Chat History(2025)http

263、s:/www.wiz.io/blog/wiz-research-uncovers-exposed-deepseek-database-leak 139 T.S.Dutta New Jailbreak Techniques Expose DeepSeek LLM Vulnerabilities,Enabling Malicious Exploits(2025)https:/ S.Schulhoff Prompt Injection vs.Jailbreaking:Whats the Difference?(2024)https:/learnprompting.org/blog/injection

264、_jailbreaking 141 https:/www.nightfall.ai/ai-security-101/data-leakage-prevention-dlp-for-llms 142 Some of the tools used are Google Cloud DLP,Microsoft Presidio,OpenAI Moderation API,Hugging Face Fine-Tuned NER Models and spaCy(links available in section 10)143 P.A.Grassi et al.,(2017)NIST Special

265、Publication 800-63-3 Digital Identity Guidelines https:/nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-63-3.pdf 144 ENISA,Basic security practices regarding passwords and online identities(2014)https:/enisa.europa.eu/sites/default/files/all_files/ENISA%20guidelines%20for%20passwords.pdf 1

266、45 Kosinski,M.,How to prevent prompt injection attacks(2024)https:/ 146 A.Peng et al.Rapid Response:Mitigating LLM Jailbreaks with a Few Examples(2024)https:/arxiv.org/abs/2411.07494 147 B.Peng et al.Jailbreaking and Mitigation of Vulnerabilities in Large Language Models(2024)https:/arxiv.org/abs/24

267、10.15236 AI Privacy Risks&Mitigations Large Language Models(LLMs)30 validation of input,filtering to detect malicious patterns,monitoring of LLMs for abnormal input behavior,implementing rate-limiting,structure queries148and limiting the amount of text a user can input149.Mitigations for Deployers:-

268、Limit the amount of sensitive data and guide users to avoid sharing unnecessary personal information through clear instructions,training and warning.Work with providers to ensure they adhere to data protection regulations and do not retain or misuse(sensitive)input data.-Require secure user authenti

269、cation to restrict access to the input interface and protect session data.As highlighted in the providers mitigations,deployers should also implement secure password practices based on the latest NIST and ENISA recommendations,150 encourage users to adopt password managers,151 and raise awareness ab

270、out secure practices and internal password policies among users,such as employees.-Clearly communicate to users how their data is handled and processed at each phase of the data flow.This could be done through(internal)privacy policies,instructions,warning or disclaimers in the user interface.-To mi

271、tigate adversarial attacks several measures can be implemented such as adding a layer for input sanitization and filtering,monitoring and logging user queries to detect unusual patterns,and incorporating post-processing layers to validate outputs.Additionally,educating users on proper usage can help

272、 reduce the likelihood of unintentional inputs that may lead to harmful outcomes.Provider interface&API Risks:Data interception:Insufficient encryption during data transmission to the providers servers may expose input to interception by third parties.API misuse:If API access is not restricted and s

273、ecured,attackers could exploit the API to intercept or manipulate data.Attackers could also overwhelm the API with excessive traffic to disrupt its availability(Denial-of-Service(DoS)Attacks).Interface vulnerabilities:Interface vulnerabilities refer to weaknesses in the providers user interface that

274、 may expose user data to malicious actors.These vulnerabilities can stem from technical flaws(e.g.,insufficient input validation,misconfigured API endpoints)or social engineering tactics such as phishing.For example,attackers could create fake versions of a chatbot interface(e.g.,replicating the des

275、ign and branding)to trick users into entering sensitive information such as credentials,payment details,or personal data.Malicious actors could develop deceptive applications claiming to be legitimate integrations with the providers API,tricking end-users into sharing sensitive data.Mitigations for

276、Providers:-Enforce end-to-end encryption for all data transmissions,regularly update encryption protocols,and use secure key management practices.-Implement strong authentication(e.g.,API keys,OAuth),enforce rate limits152,monitor for suspicious activity(anomaly detection),and regularly audit API se

277、curity.-Perform regular security testing,apply input validation to prevent attacks,and implement robust session management controls.Regarding phishing,both provider 148 S.Cheng et al.StruQ:Defending Against Prompt Injection with Structured Queries(2024)https:/arxiv.org/abs/2402.06363 149 Open AI Pla

278、tform,Safety best practices(n.d)https:/ 150 Trust Community,NIST password guidelines 2025:15 rules to follow(2024)https:/community.trustcloud.ai/article/nist-password-guidelines-2025-15-rules-to-follow/151 Wikipedia,Password Manager(2025)https:/en.wikipedia.org/wiki/Password_manager 152 LLM Engine,(

279、n.d)https:/llm- Privacy Risks&Mitigations Large Language Models(LLMs)31 and deployer have roles in addressing this risk.The provider should implement platform-level protections,such as safeguarding the authenticity of their interface(e.g.,anti-spoofing measures,branding protections,secure APIs),moni

280、toring for suspicious activity,and providing tools to help deployers detect and prevent abuse.Mitigations for Deployers:-If the deployer is using only the providers interface,their responsibility is limited to securely managing access credentials and complying with data handling policies;however,if

281、the deployer integrates the providers API into their own systems,they are additionally responsible for securing the integration,including encryption,monitoring,and safeguarding data in transit.Both provider and deployer should design,develop,deploy and test applications and APIs in accordance with l

282、eading industry standards(e.g.,OWASP for web applications153)and adhere to applicable legal,statutory or regulatory compliance obligations.-The deployer should educate employees and end users about evolving phishing techniquessuch as fake interfaces,deceptive emails,or fraudulent integrationsthat co

283、uld trick individuals into revealing sensitive information.Education should focus on recognizing suspicious behaviors and verifying the legitimacy of communications and interfaces.LLM processing at Providers infrastructure Risks:Model inference risks:During processing,the model might inadvertently i

284、nfer sensitive or inappropriate outputs based on the training data or provided input.(Un)intended data logging:Providers can log user input queries and outputs for debugging or model improvement,potentially storing sensitive data154 without explicit user consent.If logged user queries are included i

285、n training data,in case of an adversarial attack,attackers might introduce malicious or misleading content to manipulate the models future outputs(data poisoning attack)155.Anonymization failures:Inadequate anonymization or filtering techniques could lead to the inclusion of identifiable user data i

286、n model training datasets,raising privacy concerns.Unauthorized access to logs:Logs containing user inputs and outputs could be accessed by unauthorized personnel or exploited in the event of a data breach.Data aggregation risks:If logs are aggregated over time,they could form a comprehensive datase

287、t that may reveal patterns about individuals,organizations,or other sensitive activities.Third-party exposure:If the provider relies on external cloud infrastructure or third-party tools for LLM processing,theres an added risk of data exposure through those dependencies.These dependencies involve ex

288、ternal systems,which may have their own vulnerabilities.Lack of data retention policies:The provider could store the data indefinitely without having retention policies in place.Mitigations for Providers:-Implement strict content filtering mechanisms and human review processes to flag sensitive or i

289、nappropriate outputs.-Minimize data logging,collect only necessary information,and ensure you have a proper legal basis for any processed data.Use trusted sources for training data and validate its quality.Sanitize and preprocess training data to eliminate vulnerabilities or biases.Regularly review

290、and audit training data and fine-tuning processes for 153 OWASP,OWASP Top Ten(2025)https:/owasp.org/www-project-top-ten/154 The data stored could be sensitive data such as credit card numbers,or special category of data such as health data(article 9 GDPR).155 Aubert,P.et al.,Data Poisoning:a threat

291、to LLMs Integrity and Security(2024)https:/www.riskinsight- Privacy Risks&Mitigations Large Language Models(LLMs)32 issues or manipulations.Implement monitoring and alerting systems to detect unusual behavior or potential data poisoning156.-Apply robust anonymization techniques,regularly test them f

292、or effectiveness,and use automated tools to identify and remove identifiable data before use in training.-Enforce strong access controls,encrypt log data,and monitor access logs for suspicious activity to prevent breaches.-Providers must implement a robust third-party risk management program,adherin

293、g to best known frameworks157 to ensure a secure environment.Key measures include conducting thorough vendor assessments,ensuring compliance with security standards,requiring strong data encryption during transmission and storage,conducting security audits,implementing real-time monitoring and incid

294、ent response plans tailored to third-party dependencies.Providers should also implement protections against threats such as DoS/DDoS attacks,which can disrupt operations and expose systems to further risks.-Clearly define retention policies,align them with legal requirements and where possible provi

295、de users with options to delete their data.Processed Output Risks:Inaccurate or sensitive responses:The model may generate outputs that reveal unintended sensitive information or provide inaccurate or misleading information(hallucinations)158,leading to harm or misinformation.Re-identification risks

296、:Outputs could inadvertently reveal information about the users query or context that can be linked back to them.Output misuse159:Users or third parties may misuse the generated output.Mitigations for Providers:-Implement post-processing filters to detect and remove sensitive or inaccurate content,a

297、nd regularly retrain the model using updated and verified datasets to improve response accuracy.Implement disclaimers to highlight potential limitations of AI-generated responses.-Apply privacy-preserving techniques to help you redact sensitive identifiers in outputs and minimize the inclusion of un

298、necessary contextual details in generated responses.-Provide clear usage policies,educate users on ethical use of outputs,and implement mechanisms to detect and prevent the misuse of generated content where feasible.Mitigations for Deployers:-For critical applications,ensure generated outputs are re

299、viewed by humans before implementation or dissemination.-Educate end-users on ethical and appropriate use of outputs,including avoiding overreliance on the model for critical or high-stakes decisions without verification.-Securely store outputs and restrict access to authorized personnel or systems

300、only.2.Data Flow in an off-the-shelf LLM System 2.Data Flow in an off-the-shelf LLM System The most common use case for this service model involves organizations leveraging a pre-trained model from a platform to develop and deploy their own AI system.Once the AI system is operational,the data 156 OW

301、ASP,LLM10:2023-Training Data Poisoning(2023)https:/owasp.org/www-project-top-10-for-large-language-model-applications/Archive/0_1_vulns/Training_Data_Poisoning.html 157 Center for Internet Security,The 18 CIS Critical Security Controls(2025)https:/www.cisecurity.org/controls/cis-controls-list 158 Wi

302、kipedia,Hallucination Artificial Intelligence(2025)https:/en.wikipedia.org/wiki/Hallucination_(artificial_intelligence)159 OWASP,LLM05:2025 Improper Output Handling(2025)https:/genai.owasp.org/llmrisk/llm052025-improper-output-handling/AI Privacy Risks&Mitigations Large Language Models(LLMs)33 flow

303、closely resembles that of an LLM as a Service model,particularly during user interactions and output generation.However,several key differences and limitations set these models apart:Roles and responsibilities:Organizations developing an LLM system using the off-the-shelf model may be considered pro

304、viders160,particularly when they intend to place the system on the market for use by others(deployers of their system and end-users).This introduces an additional layer of responsibility for data handling,security,and compliance with privacy regulations.The organization may also be developing the AI

305、 system for its own internal use.Hosting and processing:In a LLM off-the-shelf based system,the provider hosts the model on their infrastructure or a third-party cloud environment of their choice.This contrasts with the LLM as a Service model,where hosting and processing are entirely managed by the

306、original model provider.The new provider is now responsible for all aspects of system integration,maintenance,and security.Customization and training:A notable difference is that the initial training and fine-tuning of the model were conducted by the original provider,which can introduce risk and li

307、mitations:o The new provider has often no oversight or knowledge of the contents of the dataset used during the models initial training,which may introduce biases,inaccuracies,or unknown privacy risks161.o The new provider remains dependent on the original provider for updates or bug fixes to the mo

308、del architecture,potentially delaying critical improvements or fixes.o Fine-tuning may be limited by the capabilities of the off-the-shelf model.New providers might only be able to adjust certain parameters or add new layers rather than fully retrain the model,restricting its adaptability for highly

309、 specific use cases.o In such cases,retrieval-augmented generation(RAG)is a commonly used alternative.Instead of embedding domain-specific knowledge into the model itself,RAG connects the model to an external knowledge base and retrieves relevant documents at runtime to ground its responses.This ena

310、bles dynamic,accurate,and updatable answers without modifying the base model,a key advantage for domains with evolving information or regulatory requirements.160 According to Article 25 of the AI Act,a deployer of a high risk AI system becomes a provider when they substantially modify an existing AI

311、 system,including by fine-tuning or adapting a pre-trained model for new applications.In such cases,the deployer assumes the responsibilities of a provider under the AI Act.161 EDPB Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of AI mod

312、els.Adopted on 17 December 2024 Figure 10.Data Flow in an off-the-shelf LLM System AI Privacy Risks&Mitigations Large Language Models(LLMs)34 A similar approach is cache-augmented generation(CAG)162 which can reduce latency,lower compute costs,and ensure consistency in responses across repeated inte

313、ractions but that is less practical for large datasets that are often updated.The figure below illustrates how RAG163 works:the users query is first enhanced with relevant information retrieved from an external database,and this enriched input is then sent to the language model to generate a more ac

314、curate and grounded response.Some common privacy risks of using RAG are:Insecure logging or caching:User queries and retrieved documents may be stored insecurely,increasing the risk of unauthorized access or data leaks.Third-party data handling:If the retrieval system uses external APIs or services,

315、user queries may be sent to third parties,where they can be logged,tracked,or stored without user consent.Exposure of sensitive data:The model may retrieve personal or confidential information if this is stored in the knowledge base.3.Data Flow in a Self-developed LLM System 3.Data Flow in a Self-de

316、veloped LLM System In a self-developed LLM system,the organization takes full responsibility for designing,training,and in some cases also deploying the model.This approach provides maximum control over the LLM model but also introduces unique challenges across the data flow.Since we have already ex

317、plored an example of privacy risks within the AI lifecycle data flow in a previous section,we will take here a more general approach,focusing on some of the important phases for this service model.The general data flow phases could be as follow:Dataset collection and preparation:The organization col

318、lects and curates164 large-scale datasets for training the LLM.Model training:The training phase involves using the collected dataset to develop the LLM.This typically requires significant computational resources and specialized infrastructure,such as high-performance GPUs or 162 Sharma,R.,Cache RAG

319、:Enhancing speed and efficiency in AI systems(2025)https:/ Theja,R.,Evaluate RAG with LlamaIndex(2023)https:/ 164 Atlan,Data Curation in Machine Learning:Ultimate Guide 2024(2023)https:/ 11.RAG Diagram Open AI Cookbook AI Privacy Risks&Mitigations Large Language Models(LLMs)35 distributed computing

320、systems.Before deployment,the model undergoes rigorous evaluation and testing using separate validation and test datasets to ensure its accuracy,reliability,and alignment with intended use cases.Fine-Tuning:After initial training,the model may be fine-tuned using additional datasets to specialize it

321、s capabilities for specific tasks or domains.Deployment:The trained and fine-tuned LLM is integrated into the organizations infrastructure,making an interface available for end-users.User input:End-users interact with the deployed AI system by submitting inputs through an interface such as an app,ch

322、atbot,or custom API.Provider interface&API:The input is sent through an interface or application.This interface ensures the input is formatted appropriately and securely transmitted to the LLM infrastructure.Model processing:The self-developed LLM processes user inputs locally or on the organization

323、s(cloud)infrastructure,generating contextually relevant responses using its trained parameters.Processed output delivery:The processed outputs are delivered to end-users or integrated into downstream systems for actionable use.Outputs may include text-based responses,insights,or recommendations.Priv

324、acy considerations in self-developed LLM systems Self-developing an LLM system provides significant control but also introduces privacy and data protection risks at each phase of the data flow.Below are some of the key risks and suggested mitigations:Phases Possible Risks&Mitigations Dataset collect

325、ion and preparation Risks:Sensitive data inclusion:Collected datasets could(inadvertently)include personal or sensitive information.Legal non-compliance:The data could be collected unlawfully violating data protection regulations like GDPR.Dataset Collection and PreparationModel Training&Fine-tuning

326、DeploymentFigure 12.Data Flow in a Self-developed LLM AI Privacy Risks&Mitigations Large Language Models(LLMs)36 Bias and discrimination:Datasets could reflect societal or historical biases,leading to discriminatory outputs.Data poisoning:Datasets may be intentionally manipulated by malicious actors

327、 during collection or preparation,introducing corrupted or adversarial data to mislead the model during training.Mitigations for Providers:-Apply anonymization and pseudonymization techniques165 to minimize privacy risks.-In certain use cases,and after carefully weighing the potential pros and cons,

328、166167 the creation of synthetic data using LLMs168 could be an alternative to the use of real personal data.However,synthetic data might not always be suitable169,as its quality and utility depend on the specific requirements and context of the application.-Ensure data collection is compliant with

329、regulations.-Regularly audit datasets for bias and sensitive content,removing any problematic entries.-Implement robust data validation and monitoring to detect and prevent malicious or corrupted data.Use trusted data sources,apply automated checks for anomalies,and cross-validate data from multiple

330、 sources.Model training Risks:Unprotected training environment:Training infrastructure may be vulnerable to unauthorized access,which could expose sensitive data or allow malicious actors to compromise the training process.Data overfitting:The model may inadvertently memorize sensitive information i

331、nstead of generalizing patterns.Mitigations for Providers:-Cybersecurity should follow a layered approach,implementing multiple defenses to prevent unauthorized access and mitigate its impact.Mitigations that could be implemented include:using secure computing environments with strong access control

332、s during training(e.g.,multi-factor authentication(MFA),privileged access management(PAM)170,or role-based access controls(RBAC)171);applying network segmentation to isolate the training infrastructure from other systems and reduce the attack surface;monitoring and logging access to promptly detect

333、and respond to unauthorized activities;and using encryption for both data at rest and in transit to secure sensitive training data.Another mitigation measure to reduce data exposure is the integration of differential privacy techniques,which add noise to training data to prevent individual data points from being re-identified,even if the model is compromised.165 ENISA,Pseudonymisation techniques a