GenAIPABench: A Benchmark for Generative AI-based Privacy Assistants

dc.contributor.authorHamid, Aamir
dc.contributor.authorReddy Samidi, Hemanth
dc.contributor.authorFinin, Tim
dc.contributor.authorPappachan, Primal
dc.contributor.authorYus, Roberto
dc.date.accessioned2023-09-27T15:35:51Z
dc.date.available2023-09-27T15:35:51Z
dc.date.issued2024
dc.descriptionProceedings on Privacy Enhancing Technologies 2024(3)
dc.description.abstractWebsite privacy policies are often lengthy and intricate. Privacy assistants assist in simplifying policies and making them more accessible and user-friendly. The emergence of generative AI (genAI) offers new opportunities to build privacy assistants that can answer users’ questions about privacy policies. However, genAI’s reliability is a concern due to its potential for producing inaccurate information. This study introduces GenAIPABench, a benchmark for evaluating Generative AI-based Privacy Assistants (GenAIPAs). GenAIPABench includes: 1) A set of curated questions about privacy policies along with annotated answers for various organizations and regulations; 2) Metrics to assess the accuracy, relevance, and consistency of responses; and 3) A tool for generating prompts to introduce privacy policies and paraphrased variants of the curated questions. We evaluated 3 leading genAI systems—ChatGPT-4, Bard, and Bing AI—using GenAIPABench to gauge their effectiveness as GenAIPAs. Our results demonstrate significant promise in genAI capabilities in the privacy domain while also highlighting challenges in managing complex queries, ensuring consistency, and verifying source accuracy.en_US
dc.description.urihttps://petsymposium.org/popets/2024/popets-2024-0081.phpen_US
dc.format.extent17 pagesen_US
dc.genrejournal articlesen_US
dc.genrepresentations (communicative events)en_US
dc.genreconference papers and proceedings
dc.identifierdoi:10.13016/m29yec-3iy3
dc.identifier.citationHamid, Aamir, Hemanth Reddy Samidi, Primal Pappachan, Tim Finin, and Roberto Yus. “GenAIPABench: A Benchmark for Generative AI-Based Privacy Assistants.” Proceedings on Privacy Enhancing Technologies, 2024. https://petsymposium.org/popets/2024/popets-2024-0081.php.
dc.identifier.urihttps://doi.org/10.56553/popets-2024-0081
dc.identifier.urihttp://hdl.handle.net/11603/29899
dc.language.isoen_USen_US
dc.publisherProceedings on Privacy Enhancing Technologies
dc.relation.isAvailableAtThe University of Maryland, Baltimore County (UMBC)
dc.relation.ispartofUMBC Computer Science and Electrical Engineering Department Collection
dc.relation.ispartofUMBC Faculty Collection
dc.relation.ispartofUMBC Student Collection
dc.rightsAttribution 4.0 International (CC BY 4.0)*
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/*
dc.subjectUMBC Ebiquity Research Group
dc.titleGenAIPABench: A Benchmark for Generative AI-based Privacy Assistantsen_US
dc.typeTexten_US
dcterms.creatorhttps://orcid.org/0000-0002-6593-1792en_US
dcterms.creatorhttps://orcid.org/0000-0002-9311-954Xen_US

Files

Original bundle

Now showing 1 - 2 of 2
Loading...
Thumbnail Image
Name:
popets-2024-0081.pdf
Size:
4.74 MB
Format:
Adobe Portable Document Format
Loading...
Thumbnail Image
Name:
1402.pdf
Size:
5.17 MB
Format:
Adobe Portable Document Format

License bundle

Now showing 1 - 1 of 1
Loading...
Thumbnail Image
Name:
license.txt
Size:
2.56 KB
Format:
Item-specific license agreed upon to submission
Description: