0
$\begingroup$

Goal

  • Form many survey answers getting a summary = (average answer of customers) without reading thousands of answers

ERROR

  • When I am running it on a Wikipedia article ex.: Assassins' creed just the History section It gives back reasonable results. No special characters and understandable statements
  • My dataset sometime people just writhe "hjkfahfbtw" as answer, not full sentences, So I closed down all answers if it was not closed down with a dot.
  • My answers are about 100'000 string long for each question. I haven't cleaned or done anything with them I just copied them to ''' in between 2x3 coma as a string''' and made it equal with a variable

Question

  • How do I get Longer +512 String long output text without this many special character results just normal English text?
  • The many special character appears mostly after 512 characters. So probably when input set above max_length=512 than it appears mostly (but I am not sure)? But the goal would be to get longer summaries like 1000 stings.
  • I don't just want to skip too long sentences with truncation=True in summarizer(text5, min_length=800, max_length=1000, truncation=True). A previosue error was solved by this but (that previouse error: "Token indices sequence length is longer than the specified maximum sequence length for this model (4746 > 512). Running this sequence through the model will result in indexing errors")

1. CODE [Assassin History Chapter]

import tensorflow as tf # VERSION CPU: 2.3.1
# transformers VESRION: 4.9.2
from transformers import pipeline

summarizer = pipeline("summarization", model="t5-base", tokenizer="t5-base", framework="tf",) 

text = "The Assassin's Creed series...WHOLE HISTORY PART OFTHE WIKIPEDIA ARTICLE...Auto Online."

summarizer(text, min_length=25, max_length=200)

summarizer(text, min_length=800, max_length=1000)

1. OUTPUT [Assassin History Chapter]

summarizer(text, min_length=25, max_length=200)

[{'summary_text': "the Assassin's Creed series originated out of ideas for a sequel for Prince of Persia: The Sands of Time, which was made for PlayStation 2, GameCube, Xbox and other platforms . the team decided on taking the gameplay from the original game into an open world approach, taking advantage of the improved processing power to render larger spaces and crowds . after Syndicate, the team realized that the series needed a major re-invention across both gameplay and narrative ."}]

summarizer(text, min_length=800, max_length=1000)

[{'summary_text': 'the Assassin's Creed series originated out of ideas for a sequel for Prince of Persia: The Sands of Time, which was made for PlayStation 2, GameCube, Xbox and other platforms . the team decided on taking the gameplay from the original game into an open world approach, taking advantage of the improved processing power to render larger spaces and crowds . after release of the first game in 2007, the team realized that the series needed a major re-invention across both gameplay and narrative . this led to . .. . also . (. - .- . . n ., - n s - n. , s -- \xad\xad\xad \xad -\xad\xad- \xad-\xad .\xad\xadn\xad\xads\xad\xad. \xad—\xad\xad...\xad\xad[n\xad s\xad n\xads-\xads \xad '- -\xad\xad\xad_\xad\xad[[[\xad[[]][[[[[\xad\xad][\xad[\xad]\xad__\xad[[\xad_—[[-[[.[._[.\xad[.—[.-———\xad——\xad——\xad-—— '\xad\xad—–\xad\xad–—\xad–\xad––\xad— \xad—\xad—\xad_\xad-—\xad_\xad_ -\xad —— — —\xad ——– —-\xad—- —–– – '— -– -— – –- –\xad –— \xad– ’ — ‘\xad \xad ’\xad ' ' _ . ‘\xad\xad, . « .— . [\xad— ‘_____________________ = . * ________________ — [ *_ = = . = = === == ' == = '=== * * * . + = = * * + == * = = + = == + = * * + & & + _ + , = _ = ,= , * , + .= _& _ * & * _= = * s = &= & = a = ; = ! = : .& ./ . and .* . 1 . 1. . 2. . 2 . 4. . 3. . 3 . 5. . 4 . 5 .; .:. ; &. && ,... and. _. ... & ( &, &; _ [ & [ _ ( .) .( .’ .s. '& - ( -& '. a .' . " ." . $ .# . # . @ .__&..&.&&&# &## '###&& ### #&# ##&#& #&& *& * && & # #& # ' #& *## * ' &# ' ( '''}]

2. CODE [MY SURVEY DATA]

import tensorflow as tf # VERSION CPU: 2.3.1
from transformers import pipeline

summarizer = pipeline("summarization", model="t5-base", tokenizer="t5-base", framework="tf",) 

survey_data = """
...ALL THE TEXT HERE IN 1 SINGLE LINE LIKE 100'000 STRING...
"""

summarizer(survey_data, min_length=400, max_length=500)

2. OUTPUT [MY SURVEY DATA]

Example without truncation=True

[{'summary_text': 'i'm not a huge fan of cup cakes, but i do enjoy them . i would like to see more of what you're doing on this site . I'd be interested in a lot more of the stuff you've been selling . it's a great place to start if you can't find a sweet cake. you'll be able to buy more of your products if they're available . but you'd have to be careful what you are doing . n - . " - - n an - " s s ' . " -. -- .- '- -n ' " '" nn '' '. 'n n' -" .. ." / """" -/ " "" "" " " / '/" s"" " "n" " ./ ...." .' n/ " n. /// /" "/""- s' ""- " " "- "- & /'""/ "' &' "' " "'" "n &/"' "n " & " i" "'' "/ &" i '& '( " e /- ""/ n" t " t /. " "//. ? " sn"// "n/ i/ "/ "-/ s/ -, " a /( / ( /n// (//(//-//'//)//"/ ( "/((/( "((("/('/ "(/" (/"(/'(/.//?"/''/' "( "/../.)"/."" (" "(. "/ ((('(( ((/)" "[[(([[['[(['([('[[.''('''[''))''-''&''"(( '[/ "[(/ (''")))""( "[' "[. "[&/.'" (( " "[] "( .) " "(" "-" ") "()" '" "& .(" &(() . (( &) 's'' (() " ( ( ( ) s((- " ' (( ') " ((& "( ( (( ( ) "'( )" ( ( "(- "(") "(')"("( ""() ((""'()((")"('"["(["[[""[("['"'[&'["'''''' [((.'( ('(&)'(-'['(.)) "[('. "([/()")"" ("((n''s) ""[ " "& "" ( " "") " "'}]

An example with summarizer(text, min_length=400, max_length=500, truncation=True)

[{'summary_text': "i'm not the target audience for non-sugar based products . it's difficult to compete with foodchain retailers already established and constantly giving away or making 1c on ice cream. i would like a more cheap option for cakes and I know that it is hard but some more cheap version of cummies . I HIGHLY recommend a dedicated effort towards artists and indie developers . so things like muffines and other sweets and bigger products as well .. . ( .s \xad\xad\xad \xad .\xad\xad. \xadn\xad\xad-\xad\xad[\xad\xad\xad\xad*\xad\xad_\xad\xad&\xad\xads\xad\xad–\xad\xad»\xad\xad?\xad\xad—\xad\xad...\xad\xad,\xad -\xad n\xad \xad \xad__\xad \xad \xad.\xad s\xad &\xad ,\xad\xadn-\xad \xad-\xad– \xad- \xad–--\xad---\xad--\xad——\xad_—\xad———\xad—\xad— —\xad–—\xad –\xad—–\xad–––\xad–\xad — –– – ‘\xad\xad’\xad\xad ‘\xad– ‘‘\xad–‘\xad\xad‘\xad—‘–\xad––‘–– ‘––’––-––—––\xad––\xad-–\xad’–\xad‘‘–‘’–‘‘‘’’–’\xad–’‘–’—–‘—–——–-—–’-–‘-–—-–-’–-‘–- ‘‘– ’– ‘’– —– .––...–– ("}]

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.