IEEE 3168-2024
$23.33
IEEE Standard for Robustness Evaluation Test Methods for a Natural Language Processing Service That Uses Machine Learning (Approved Draft)
| Published By | Publication Date | Number of Pages |
| IEEE | 2024 | 29 |
New IEEE Standard – Active. The natural language processing (NLP) services using machine learning have rich applications in solving various tasks and have been widely deployed and used, usually accessible by application programming interface (API) calls. The robustness of the NLP services is challenged by various well-known general corruptions and adversarial attacks. Inadvertent or random deletion, addition, or repetition of characters or words are examples of general corruptions. Adversarial characters, words, or sentence samples are generated by adversarial attacks, causing the models underpinning the NLP services to produce incorrect results. A method for quantitatively evaluating the robustness the NLP services is proposed by this standard. Under the method, different cases the evaluation needs to perform against are specified. Robustness metrics and their calculation are defined. With the standard, understanding of the robustness of the services can be developed by the service stakeholders including the service developer, service providers, and service users. The evaluation can be performed during various phases in the life cycle of the NLP services, the testing phase, in the validation phase, after deployment, and so forth.
PDF Catalog
| PDF Pages | PDF Title |
|---|---|
| 1 | IEEE Std 3168™-2024 Front cover |
| 2 | Title page |
| 4 | Important Notices and Disclaimers Concerning IEEE Standards Documents |
| 8 | Participants |
| 9 | Introduction |
| 10 | Contents |
| 11 | 1. Overview 1.1 Scope 1.2 Purpose 1.3 Word usage |
| 12 | 2. Normative references 3. Definitions, acronyms, and abbreviations 3.1 Definitions 3.2 Acronyms and abbreviations |
| 13 | 4. Evaluation target 5. Evaluation cases for NLP services 5.1 Overview of evaluation cases |
| 14 | 5.2 General corruptions |
| 15 | 5.3 Adversarial attacks 6. Robustness metrics of NLP services 6.1 Metrics overview |
| 16 | 6.2 Utility metrics 6.3 Corruption resistant metrics 6.4 Adversarial resistant metrics |
| 17 | 6.5 Quality metrics 6.6 Metrics calculation for NLP services |
| 24 | 7. Test cases 7.1 Test cases for utility metrics |
| 25 | 7.2 Test cases for general corruption 7.3 Test cases for adversarial attacks |
| 27 | Annex A (Informative) Defense against adversarial attacks |
| 28 | Annex B (Informative) Bibliography |
| 29 | Back cover |
Related products
-
IEEE 802.3-2018(Redline)
IEEE Standard for Ethernet (Redline) Published By Publication Date Number of Pages IEEE 2018
-
IEEE 802.3-2022(Redline)
IEEE Standard for Ethernet (Redline) Published By Publication Date Number of Pages IEEE 2022 7025
-
ISO/IEC/IEEE 8802-11:2012/Amd 2:2014
Information technology — Telecommunications and information exchange between systems — Local and metropolitan area networks…
-
IEEE 8802-11:2012/Amd.2:-2014
ISO/IEC/IEEE International Standard for Information technology–Telecommunications and information exchange between systems Local and metropolitan area…
-
IEEE P2933 2024
IEEE Approved Draft Standard for Clinical Internet of Things (IoT) Data and Device Interoperability with…
-
IEEE 802.3 2018
IEEE Standard for Ethernet Published By Publication Date Number of Pages IEEE 2018
-
IEEE 1003.1-2008
IEEE Standard for Information Technology – Portable Operating System Interface (POSIX(TM)) Published By Publication Date…
-
IEEE 802.3-2018
IEEE Standard for Ethernet (Superseded Redline) Published By Publication Date Number of Pages IEEE 2018…
-
IEEE P3161:2022 Edition
IEEE Approved Draft Standard for Digital Retina Systems Published By Publication Date Number of Pages…
-
IEEE 802.11-2020(Redline)
IEEE Standard for Information Technology–Telecommunications and Information Exchange between Systems – Local and Metropolitan Area…
-
IEEE 802.3-2022
IEEE Standard for Ethernet Published By Publication Date Number of Pages IEEE 2022
-
IEEE ISO IEC 8802 11:2012 Amd.2: 2014
ISO/IEC/IEEE International Standard for Information technology–Telecommunications and information exchange between systems Local and metropolitan area…
-
IEEE 487.3a-2022
IEEE Standard for the Electrical Protection of Communication Facilities Serving Electric Supply Locations Through the…
-
IEEE 802.3-2022
IEEE Standard for Ethernet Published By Publication Date Number of Pages IEEE 2022 7025
-
IEEE 1839-2014
NACE International and IEEE Joint Standard Practice for Below-Grade Corrosion Control of Transmission, Distribution, and…
-
IEEE 802.3ck-2022
IEEE Standard for Ethernet Amendment 4: Physical Layer Specifications and Management Parameters for 100 Gb/s,…
-
IEEE 802.3-2022(Redline)
IEEE Standard for Ethernet (Redline) Published By Publication Date Number of Pages IEEE 2022 7025
-
IEEE 287.1-2021
IEEE Standard for Precision Coaxial Connectors at RF, Microwave, and Millimeter-Wave Frequencies–Part 1: General Requirements,…
-
IEEE 3168-2024
IEEE Standard for Robustness Evaluation Test Methods for a Natural Language Processing Service That Uses…
-
IEEE 802.11-2020
IEEE Standard for Information Technology–Telecommunications and Information Exchange between Systems – Local and Metropolitan Area…
-
IEEE C37.100.7-2023
IEEE Guide for the Evaluation of Performance Characteristics of Non-Sulfur Hexafluoride Insulation and Arc Quenching…
-
IEEE 11073-10206-2022
IEEE Standard – Health informatics — Device interoperability — Part 10206: Personal health device communication…
-
IEEE 802.3-2018
IEEE Standard for Ethernet (Superseded Redline) Published By Publication Date Number of Pages IEEE 2018
-
IEEE 802.3bj-2014
IEEE Standard for Ethernet Amendment 2: Physical Layer Specifications and Management Parameters for 100 Gb/s…
-
IEEE 1363a 2004
IEEE Standard Specifications for Public-Key Cryptography – Amendment 1: Additional Techniques Published By Publication Date…
-
IEEE 802.11bd-2022
IEEE Standard for Information Technology–Telecommunications and Information Exchange between Systems Local and Metropolitan Area Networks–Specific…
-
IEEE 802.3bj 2014
IEEE Approved Draft Standard for Ethernet Amendment 2: Physical Layer Specifications and Management Parameters for…
-
IEEE 802.3cs-2022
IEEE Standard for Ethernet Amendment 2: Physical Layers and Management Parameters for Increased-Reach Point-to-Multipoint Ethernet…
-
IEEE 802.3cs-2022
IEEE Standard for Ethernet Amendment 2: Physical Layers and Management Parameters for Increased-Reach Point-to-Multipoint Ethernet…
-
IEEE 802.1ABdh-2021
IEEE Standard for Local and metropolitan area networks– Station and Media Access Control Connectivity Discovery…
-
IEEE 802.3-2022
IEEE Standard for Ethernet Published By Publication Date Number of Pages IEEE 2022
-
IEEE 3168-2024
IEEE Standard for Robustness Evaluation Test Methods for a Natural Language Processing Service That Uses…
-
IEEE 802.11-2020
IEEE Standard for Information Technology–Telecommunications and Information Exchange between Systems – Local and Metropolitan Area…
-
IEEE 3168-2024
IEEE Standard for Robustness Evaluation Test Methods for a Natural Language Processing Service That Uses…
-
IEEE 3161-2022
IEEE Standard for Digital Retina Systems (Published) Published By Publication Date Number of Pages IEEE…
-
IEEE ANSI PC63.4 May2014 APPROVEDDRAFT
Approved Draft American National Standard for Methods of Measurement of Radio – Noise Emissions from…
-
IEEE 802.3cs-2022
IEEE Standard for Ethernet Amendment 2: Physical Layers and Management Parameters for Increased-Reach Point-to-Multipoint Ethernet…
-
IEEE 3168-2024
IEEE Standard for Robustness Evaluation Test Methods for a Natural Language Processing Service That Uses…
-
IEEE 1500-2022
IEEE Standard Testability Method for Embedded Core-based Integrated Circuits Published By Publication Date Number of…
-
IEEE 802.3-2022(Redline)
IEEE Standard for Ethernet (Redline) Published By Publication Date Number of Pages IEEE 2022