Performance of independent audits

Audit opinions referred to in Article 37(4), point (g), of that Regulation (EU) 2022/2065 should be asserted by auditing organisations with a reasonable level of assurance. To reach a reasonable level of assurance, the auditing organisation should have a high, but not absolute, level of confidence that there have been no misstatements, such as omissions, misrepresentations, or errors, which were not detected in the audit. To ensure that level of assurance, the auditing organisation should, amongst others, obtain sufficient evidence and use appropriate auditing methodologies in its assessment.

Pursuant to Article 37(1) of Regulation (EU) 2022/2065, independent audits should be conducted at least annually, aligned with the yearly cycle of risk assessments referred to in Article 34 of that Regulation. However, more frequent audits may be necessary in certain cases. The sequencing of audits should ensure a continuum of supervision for the audited providers’ compliance with Regulation (EU) 2022/2065 and relevant codes of conduct and crisis protocols. The audited provider should ensure that the period for which a given audit assesses compliance with the audited obligations and commitments complements the period covered by the previous audit of the provider’s compliance with those obligations and commitments and starts at the latest where the period covered in the previous audit has ended. As the conclusion of an audit includes both the assessment done by the auditing organisation and the establishment of an audit report, audited providers should ensure that the duration of the audit allows for audits to be concluded at least once per year and the submission of the audit reports to the Commission and the Digital Service Coordinator follows without undue delay, pursuant to Article 42(4) of Regulation (EU) 2022/2065.

While audited providers should in no circumstance interfere with the performance of the audit and its conclusions, they should fulfil their obligations under Article 37 of Regulation (EU) 2022/2065 including by agreeing contractual terms with the auditing organisation and verifying, prior to selecting an auditing organisation, that that organisation fulfils the conditions laid down in Article 37(3) of Regulation (EU) 2022/2065.

The audited provider should, for example, assess contracts it has previously concluded with the auditing organisation or contracts concluded between the auditing organisation and legal persons connected to the audited provider. The audited provider should also include clauses in contracts with auditing organisations to guarantee the respect of the conditions laid down in Article 37(3) of Regulation (EU) 2022/2065. Where the auditing organisation consists of several entities, the audited provider should ascertain that all those entities fulfil those conditions, including, where applicable, any sub-contractors hired by the auditing organisation for the purpose of supporting the performance of the audit in any way. Whereas each entity performing the audit should individually fulfil the independence requirements and the requirements on no conflict of interests, those entities should fulfil jointly the requirements related to competence, expertise, or technical resources, thereby allowing different entities to perform different parts of the audit and contribute with the capabilities, competence and expertise needed to perform the audit. The audit report should specify the responsibility of each of those entities for the respective parts of the audit.

Pursuant to Article 37(3), point (a)(i) of Regulation (EU) 2022/2065, the audited provider should pay particular attention to avoid that the auditing organisation provides non-audit services to the audited provider when verifying whether an auditing organisation fulfils the independence requirements and the requirements on no conflict of interests. The audited provider should, for example, assess whether services were provided, such as those linked to any system, software or process involved in matters relevant to the audited obligation or commitment, such as consultancy services for assessments of performance, of governance and of software, training services, development or maintenance of systems, or subcontracting content moderation. Such services also include services provided to the audited provider which consist in consulting on, or developing internal controls, or assessing, for internal purposes, the audited provider’s compliance with Regulation (EU) 2022/2065 or codes of conduct and crisis protocols, including when this is limited to punctual tests, such as third-party tests on the performance of content moderation systems. This should not exclude auditing organisations who have performed statutory financial audits.

Given the complexity and particular nature of compliance audits for Regulation (EU) 2022/2065, the subject-matter expertise of the auditing organisation is key for performing audits with a reasonable level of assurance and for exercising the professional judgment and scepticism that enables that organisation to know, for example, which information it needs to perform the audit procedures or to challenge contradictory information. The audited provider should therefore verify whether the auditing organisation provides such expertise, including in the area of risk management, both with regard to audit risks and the subject-matter of Regulation (EU) 2022/2065 and, in particular, the systemic societal risks referred to in Article 34 of that Regulation. In addition, the audited provider should verify the technical competence and capabilities of the auditing organisation in view of the specific audited service, including its subject-matter expertise, for example as regards the functioning and effects of algorithmic systems, such as recommender systems and other socio-technical systems maintained by the provider. The auditing organisation should have the possibility to subcontract or otherwise obtain and deploy the necessary expertise and capabilities, and the audited provider should verify and ensure that the auditing organisation is able to acquire that expertise and those capabilities in time for the performance of the audit.

In verifying that auditing organisations fulfil the conditions laid down in Article 37(3) of Regulation (EU) 2022/2065, the audited provider should assess relevant evidence, including, as appropriate, certifications, declarations and audit reports issued by the auditing organisation. The appropriate expertise could be proven, for example, by practical experiences in assessing and managing risks, as well as by academic activity, scientific publications, and experience with relevant audits. Audit reports should contain all the relevant supporting documentation attesting that the auditing organisation fulfils the necessary conditions.

Pursuant to Article 37(2) of Regulation (EU) 2022/2065, the audited provider is to afford all the necessary cooperation and assistance for the auditing organisation to conduct the audit in an effective, efficient, and timely manner, as well as refrain from interfering in any way with independent decisions of the auditing organisation. For example, the audited provider should not impose, give any guidance, or otherwise influence the auditing organisation through any kind of contractual or other limitations or incentives in their choice and execution of audit procedures, methodologies, collection and processing of information and audit evidence, analysis, tests, audit opinion or elaboration of audit conclusions.

To guarantee the necessary cooperation and assistance during the audit, the audited provider should ensure that the auditing organisation is granted access to all information necessary for the performance of the audit. The audited provider should send, as early as possible and in any case before the auditing organisation starts performing audit procedures, all the necessary documents and explanations. For example, pursuant to Article 41(3), points (d) and (e), of Regulation (EU) 2022/2065, the compliance function of the audited provider is to monitor compliance with all audited obligations and commitments, which should result in the elaboration of internal controls. The auditing organisation should therefore be granted access to all information related to such controls, and any other information outlining the audited provider’s strategy to ensure compliance. In particular, the audited provider should make available to the auditing organisation the benchmarks it relies upon to ensure compliance with Regulation (EU) 2022/2065 so that the auditing organisation can base the audit criteria on this information. In addition, the auditing organisation should be granted access to any analysis the audited provider might have conducted on inherent risks and control risks. That provider should make available to the auditing organisation information that facilitates the understanding of the audited service, its governance, the competence of respective teams and decision-making structures, including its compliance function, as well as presentations of its information technology (IT) systems, data and records structures, and the interplay between different algorithmic systems of relevance to the audit.

The auditing organisation should be able to request, at any time in the performance of the audit, any other necessary information. Access to that information should be granted without undue delay in a manner that does not in any way hamper the performance of the audit. This should include access to data, including personal data, collected from various sources, such as documents, algorithmic systems, databases or interviews, as appropriate. The audited provider should also grant the auditing organisation access to procedures and processes, IT systems, such as algorithmic and information systems, including testing environments. In order to allow the auditing organisation to meaningfully inspect such systems, the audited provider should make all necessary resources available to assist that organisation in accessing and assessing the systems, such as by making available the provider’s competent personnel to answer questions or operate test environments and provide explanations about their functioning, or facilitate any other necessary access to personnel and premises, such as buildings. Access to procedures and processes could imply, for example, access to descriptions or documents concerning the audited provider’s internal decision-making process. Access to relevant information may also require other ancillary actions from the audited provider to fulfil their obligation for cooperation and assistance. For example, interviews with personnel may require secure facilities provided by the audited provider. Where it is necessary for the performance of the audit, audited providers should fulfil the cooperation and assistance obligation towards auditing organisation among other things by facilitating access to relevant data related to their operations held by their third-party contractors. This might be the case, for example, as regards results of content moderation actions, training material or guidance used by third-party contractors who moderate content, or vendors and service providers for IT solutions, including, for example algorithms and applications used in recommender systems or advertising systems used by the audited provider.

To facilitate meaningful transparency of the audit findings and to provide a comprehensive and comparable format of audit reports, referred to in Article 37(4) of Regulation (EU) 2022/2065 and audit implementation reports referred to in Article 37(6) of that Regulation, this Regulation should establish templates for those reports and require a number of annexes for each of the reports. While the templates laid down in this Regulation require comprehensive reporting, they should not affect the requirements on the publication of reports provided for in Article 42(4) and (5) of Regulation (EU) 2022/2065.

In order to ensure that the auditing organisation receives all the necessary assistance from the audited provider, without interference in the performance of the audit, and that the auditing organisation fulfils all conditions for the preparation of the audit and delivers the audit report in due time and with the quality necessary to reach a reasonable level of assurance, certain rules should specify the procedures for the preparation of the audit. The duties and responsibilities of the audited provider and the auditing organisation, including all sub-contractors or partner organisations and the staff responsible for carrying out the audit, should be set out in a written agreement, including through contractual terms. The written agreement should also specify the audited obligations and commitments, the allocation of resources, and the rules of interaction and contact points between the auditing organisation and the audited provider. All supporting documentation and contracts should be annexed to the audit report, including when the documents take the form of an audit engagement letter or other contractual terms.

In order to provide a comprehensive overview and facilitate the accountability of audited providers, the audit report should include a conclusion of the auditing organisation’s assessment of compliance of the audited provider with each audited obligation or commitment. Each audit conclusion should be based on a reasonable level of assurance and should be either ‘positive’, ‘positive with comments’ or ‘negative’, in order to appropriately inform the audit opinion. Conclusions that are ‘positive with comments’ should not concern the assessment of compliance itself. Such comments could refer, for example, to the production of information by the provider at the request of the auditing organisation, or to improvements in the maintenance or controls put in place by the audited provider, or take note of further mitigation plans and improvements that the provider intends to make. In any event, where the auditing organisation deems the audited provider compliant with an audited obligation or commitment according to the benchmarks reported by the audited provider, but considers it necessary to include remarks on those benchmarks, the audit conclusion should be ‘positive with comments’, since such comments could usefully inform the provider about opportunities for potential changes to its benchmarks, based on the auditing organisation’s knowledge and expertise, as well as information from external sources. For example, the comments could be informed by guidance from the Commission, including through guidelines from the Commission referred to in Article 35(3) of Regulation (EU) 2022/2065, and any other relevant guidelines issued by the Commission with respect to the application of that Regulation, reports from the European Board for Digital Services referred to in Article 35(2) of that Regulation, enforcement actions, decisions taken by the Commission pursuant to that Regulation, relevant case law, especially from the Court of Justice of the European Union, public consultations, or relevant authoritative sources.

In order to enable public scrutiny and regulatory supervision, where an audit conclusion is ‘negative’ but applies only for a limited period of time and the auditing organisation deems that the audited provider complied with the obligation or commitment for the rest of the audited period, this should be reflected in the audit report for each concerned obligation or commitment. The report should include the auditing organisation’s observations on any information made available to them by the audited provider as regards mitigation plans in place or planned to remedy non-compliance.

In light of the different nature of the legal obligations laid down in Chapter III of Regulation (EU) 2022/2065 and the voluntary commitments made under codes of conduct and crisis protocols pursuant to Articles 45, 46 and 48 of that Regulation, the auditing organisation should issue audit opinions on the compliance with that Chapter and with each code and protocol.

In order to perform the audit with a reasonable level of assurance and to design the appropriate audit procedures according to methodologies that minimise the audit risk to a low level, a key part of the methodology for performing the audit should be the estimation of the audit risks, namely the risk that the auditing organisation expresses an inappropriate audit opinion or conclusion. Therefore, the auditing organisation should assess the audit risk at the very beginning of the audit, before designing the precise methodology and performing audit procedures. The audit risk analysis is necessary to allow the auditing organisation to select the precise methodologies for the audit and determine how comprehensive the audit procedures must be so as to attain the reasonable level of assurance for the audit opinion. The auditing organisation should perform the audit risk analysis for the assessment of compliance with each audited obligation or commitment, considering inherent risks, control risks and detection risks.

In order to correctly evaluate the audit risks, the audit risk analysis should take into account the nature of the audited service, notably its risk profile, and the scope and complexity of the audit. For example, it is likely that online platforms that allow the conclusions of distance contracts between consumers will have different inherent risks than video-sharing platforms or search engines. Furthermore, the societal and the economic context in which the audited service is provided should be considered, for example as regards typical user groups such as minors, or frequent behaviour such as a high incidence of inauthentic use and coordinated behaviours in disinformation campaigns. The societal and the economic context to be considered should also include the probability and, independently, the severity of exposure to crisis situations and unexpected events, as referred to in Regulation (EU) 2022/2065.

In order to ensure that the audit risk analysis reflects the evolution of the risks to which the service is subject, the audit risk analysis should also be based on information from previous audits to which the audited provider was subject, where applicable, and build upon information from sources such as audit reports of other providers with a similar risk profile. To ensure that the audit risk analysis is fully informed by state-of-the-art evidence of risks in contexts similar to those in which the audited provider operates, and by authoritative sources of direct relevance for the application of Regulation (EU) 2022/2065, the analysis should also rely on information from reports issued by the European Board for Digital Services or guidelines from the Commission, where applicable. Other information could also include information from audit reports published pursuant to Article 42(4) of Regulation (EU) 2022/2065 by other providers of very large online platforms or of very large online search engines.

The auditing organisation should draw up, without influence from the audited provider, the audit methodologies used to assess compliance with the audited obligations and commitments. The audit criteria should be based on the information submitted by the audited provider as regards benchmarks used by the audited provider for monitoring compliance. The methodology may also take into consideration other information made available by the audited provider, such as the analysis of inherent risks, when the audited provider has done such analysis, for example through measures developed by the compliance officer or the management body in accordance with Article 41 of Regulation (EU) 2022/2065 or other measures embedded in the functioning of the service for the systemic risk assessments referred to in Article 34 of that Regulation.

In order to ensure that the audit methodologies are appropriate for reaching a reasonable level of assurance for the audit opinions, the choice of methodology for the audit procedures should address the specificities of the audited obligation or commitment and should be adapted, for example, to the nature of the audited obligation as an obligation of means, or an obligation of results that the provider must achieve in order to be compliant. For example, auditing procedures for assessing compliance with transparency reporting pursuant to Article 15 of Regulation (EU) 2022/2065 could allow the auditing organisation to conclude whether the reports were published within the delays and formats requested in that Regulation, as well as whether they were complete and the data reported was accurate, representative, and appropriately broken down, for example, per category of illegal content actioned.

The choice of methodology should also depend on whether the compliance assessment requires contextual interpretations by the auditing organisation. The selection of methodologies should also be adapted to the inherent risks linked to the activities carried out in providing the service and the context in which the service is provided, for example, whether the service involves the sale of goods that could be illegal or whether the service is primarily used by minors. For example, methodologies for assessing compliance with obligations to put in place appropriate and proportionate measures to ensure a high level of privacy, safety, and security of minors pursuant to Article 28(1) of Regulation (EU) 2022/2065 should allow the auditing organisation to have a sufficient understanding of how the audited service is used by minors and the risks to their privacy, safety and security that may be incurred, as well as what constitutes an appropriate and proportionate measure in the specific context of the audited service and its use by minors. To this end, auditing organisations should break down the assessment into appropriate steps. They should assess the audit risks according to the risk profile of the audited provider, notably whether it is available to or predominantly used by minors. They should assess, for example, whether the provider has put in place age assurance tools, whether these are effective and how the audited provider assesses and monitors their effectiveness. They should assess whether the audited provider has put in place appropriate measures for detecting adversarial use of their service and behavioural patterns that seek to harm or are likely to harm minors.

The selection of methodologies should be adapted to the control risks linked to the compliance measures put in place by the audited provider, as well as to the detection risks, namely the risk not to detect misstatements in the information the provider makes available to the auditing organisation. For example, where an audited obligation could involve the audit of an algorithmic system based on personalisation for individual recipients of the audited service and on recurrent updates of the algorithmic system, such as the disclosure obligations for recommender systems pursuant to Article 27 of Regulation (EU) 2022/2065, the choice of methodology should allow the auditing organisation to design the appropriate tests to minimise detection risks. Similarly, where the auditing organisation seeks to assess whether all relevant risks were mitigated in the design, functioning and use of applications based on large-scale language models, such as chat functionalities or recommender systems deployed by the audited provider, the auditing organisation should first assess the appropriateness of the controls put in place by the provider. The choice of tests should be informed by the robustness of those internal controls. In particular, but not limited to the cases where the internal controls are weak, incomplete or inconclusive to assess whether the rules are complied with when considering the population of recipients of the audited service, the audit procedures should rely on a combination of methodologies. For example, methodologies could include substantive analytical procedures, such as the analysis of the interactions between all algorithmic systems involved in the recommender systems and related decision-making rules and processes for establishing the main parameters of those recommender systems, observations of digital records and logs. Methodologies should also include tests of the system, such as tests in simulated environments.

In order to ensure that the methodology is relevant and adapted to new findings during the performance of the audit, the selection of methodologies should be guided by the professional judgment of the auditing organisation and should be adjusted to address those new findings, in particular when the auditing organisation has reasonable doubts in relation to the information submitted by the audited provider. The professional scepticism of the auditing organisation should be based on its expertise, as well as on other sources of information of particular relevance for the application of Regulation (EU) 2022/2065 such as reports from the European Board for Digital Services, guidance from the Commission, audit reports issued from codes of conduct or crisis protocols referred to in Articles 45, 46 and 48 of that Regulation or information emerging during the performance of the audit, including when related to events, in particular crisis situations, that require additional actions from the audited provider to ensure compliance with certain audited obligations or commitments.

In order to ensure that sufficient audit evidence is gathered during the audit, auditing organisations should assess both the internal controls of the audited provider and perform substantive audit procedures for assessing the audited provider’s compliance. In certain cases, the auditing organisation should also perform tests.

Given the complexity of algorithmic systems used by providers of online platforms and their important role in complying with several obligations laid out in Regulation (EU) 2022/2065, particular attention should be paid to the necessary and appropriate methodological choices for auditing algorithmic systems. This is the case both when algorithmic systems are part of the controls put in place by the audited provider, and when they are themselves the subject-matter of the audited obligations or commitments, such as with respect to recommender systems, for example pursuant to Articles 27, 34, 35, and 38 of Regulation (EU) 2022/2065, or advertising systems, for example pursuant to Articles 26, 28, 34, 35, and 39 of that Regulation, content moderation systems, for example pursuant to Articles 14, 15, 34 and 35 of that Regulation, or any other algorithmic system that contributes to the risks referred to in Article 34 of that Regulation.

A combination of substantive analytical procedures should also be used, including, as appropriate, based on observations of processes and activities of the audited provider in designing, developing, operating, testing, and monitoring algorithmic systems, or observations of digital records and logs produced by the systems. The methodologies should be adapted to the particularities of algorithmic systems, including their governance, the interaction between different algorithmic systems as well as the related data management systems, and to the technologies underlying those algorithmic systems, such as generative models or other classifiers, selection or search algorithms.

Furthermore, audit methodologies for algorithmic systems should include tests, for example, to gather information that the audited provider has not previously documented, or to independently reproduce and assess results of accuracy indicators, tests in sandboxes or simulated environments, or tests in production systems, including through data scraping or adversarial testing.

Given that the high quality of the audit evidence is a necessary condition for an auditing organisation to form an audit opinion with a reasonable level of assurance, the information that the auditing organisation decides to use as audit evidence should be appropriate and sufficient to reduce audit risks. In addition, the audit evidence should be reliable according to the auditing organisation’s professional judgment and scepticism and, where appropriate, in the light of alternative sources of information. Professional judgment and scepticism should include a critical assessment of audit evidence and possible misstatements. Those quality standards should apply to all audit evidence regardless of whether it has been provided by the audited provider or collected from other sources.

A range of sources of information should be considered by the auditing organisation and could include, for example, interviews with the personnel or contractors of the audited provider, including compliance officers, engineers, data scientists, software architects, or members of internal audit teams. They could also include technical documentation on the design, implementation, testing and monitoring of a relevant system, including on data quality and governance and on updates and versions of the system, and other documents on the audited provider’s governance and decision-making processes, including in view of priorities, resources, allocations of tasks and responsibilities, or the expertise of relevant personnel.

In order to ensure efficiency and proportionality in the performance of the audit, the auditing organisation should be allowed to sample data and information, with due regard to reaching a representative sample, to enable the auditing organisation to reach an audit opinion with a reasonable level of assurance. To ensure transparency and reproducibility of the audit procedures, the auditing organisation should provide justifications of the choices of sample size and method of the sampling in the audit report. For example, the size of the sample and methodology should be selected considering what is effective in meeting the purpose of auditing the specific audited obligation or commitment, and to minimise the risk that the conclusion of auditing the specific sample is different from what the conclusion would be if the entire population of evidence were subjected to the auditing procedure. The size and methodology for the sample should be selected considering the full scope of the audit, as well as internal or external changes to the audited service during this time. They should also be adapted to the particularities of algorithmic systems including as regards personalisation by profiling. As a part of this consideration the auditing organisation should, for example, appropriately sample from the different cohorts or partitions that may result from personalisation techniques, or identify the margin of error and justify why it is at an acceptable level.

Given the novelty of certain provisions of Regulation (EU) 2022/2065, it is necessary to set out methodological principles, including audit questions, and further orientations for the selection of the audit methodologies and audit evidence for the assessment of compliance with those provisions, namely for assessing compliance with Articles 34, 35 and 36 of Regulation (EU) 2022/2065 on the performance of risk assessments and the adoption of risk mitigation measures by audited providers and on the application of obligations with respect to crisis response.

Given that auditing organisations should also assess compliance of the audited provider with Article 37 of Regulation (EU) 2022/2065, further specifications should also be provided on the precise audit with respect to which compliance should be assessed, in particular to avoid any conflicts of interests for the auditing organisation.

In view of the voluntary nature of codes of conduct and crisis protocols, it is necessary to provide specific rules for auditing compliance with Articles 45, 46 and 48 of Regulation (EU) 2022/2065, in particular to ensure that auditing organisations dispose of all the necessary information to perform audits specific to the commitments under each code of conduct and crisis protocol,