I don’t think AI is actually that good at summarizing.
It really depends on the type and size of text you want it to summarize.
For instance, it’ll only give you a very, very simplistic overview of a large research paper that uses technical terms, but if you want to to compress down a bullet point list, or take one paragraph and turn it into some bullet points, it’ll usually do that without any issues.
Edit: I truly don’t understand why I’m getting downvoted for this. LLMs are actually relatively good at summarizing small, low-context-necessary pieces of information into bullet points. They’re quite literally made as code that interprets the likelihood of text based on an input. Giving it a small amount of text to rewrite or recontextualize is one of its best strengths. That’s why it was originally mostly implemented as a tool to reword small isolated sections in articles, emails, and papers, before the technology was improved.
It’s when they get to larger pieces of information, like meetings, books, wikipedia articles, etc, that they begin to break down, due to the nature of the technology itself. (context windows, lack of external resources that humans are able to integrate into their writing, but LLMs can’t fully incorporate on the same level)
But if the text you’re working on is small, you could just do it yourself. You don’t need an expensive guessing machine.
Like, if I built a rube-goldberg machine using twenty rubber ducks, a diesel engine, and a blender to tie my shoes, and it gets it right most of the time, that’s impressive. but also kind of a stupid waste, because I could’ve just tied them with my hands.
It can, but I don’t see that happen often in most places I see it used, at least by the average person, although I will say I’ve deliberately insulated myself a bit from the very AI bro type of people who use it regularly throughout their day, and mostly interact with people who are using it occasionally during research for an assignment, rewriting part of their email, etc, so I recognize that my opinion here might just be influenced by the type of uses I personally see it used for.
In my experience, when it’s used to summarize, say, 4-6 sentences of text, in a general-audience readable text (i.e. not a research paper in a journal) that doesn’t explicitly rely on a high level of context from the rest of the text (e.g. a news article relies on information it doesn’t currently have, so a paragraph out of context would be bad, vs instructions on how to use a tool, which are general knowledge) then it seems to do pretty well, especially within the confines of an existing conversation about the topic where the intent and context has been established already.
For example, a couple months back, I was having a hard time understanding subnetting, but I decided to give it a shot, and by giving it a bit of context on what was tripping me up, it was successfully able to reword and re-explain the topic in such a way that I was able to better understand it, and could then continue researching it.
Broad topic that’s definitely in the training data + doesn’t rely on lots of extra context for the specific example = reasonably good output.
But again, I also don’t frequently interact with the kind of people that like having AI in everything, and am mostly just around very casual users that don’t use it for anything very high stakes or complex, and I’m quite sure that anything more than extremely simple summaries of basic information or very well-known topics would probably have a lot of hallucinations.
See, when I have 4-6 sentences to summarize, I don’t see the value-add of a machine doing the summarizing for me.
Oh I completely understand, I don’t often see it as useful either. I’m just saying that a lot of people I see using LLMs occasionally are usually just shortening their own replies to things, converting a text based list of steps to a numbered list for readability, or just rewording a concept because the original writer didn’t word it in a way their brain could process well, etc.
Things that don’t necessarily require a huge amount of effort on their part, but still save them a little bit of time, which in my conversations with them, seems to prove valuable to them, even if it’s in a small way.
I feel like letting your skills in reading and communicating in writing atrophy is a poor choice. And skills do atrophy without use. I used to be able to read a book and write an essay critically analyzing it. If I tried to do that now, it would be a rough start.
I don’t think people are going to just up and forget how to write, but I do think they’ll get even worse at it if they don’t do it.
Our plant manager likes to use it to summarize meetings (Copilot).
It in fact does not summarize to a bullet point list in any useful way.
Breakes the notes into a headers for each topic then bullet points
The header is a brief summary. The bullet points? The exact same summary but now broken by sentences as individual points.
Truly stunning work. Even better with a “Please review the meeting transcript yourself as AI might not be 100% accurate” disclaimer.
Truely worthless.
That being said, I’ve a few vision systems using an “AI” to recognize product that doesn’t meet the pre taught pattern. It’s very good at this
This is precisely why I don’t think anybody should be using it for meeting summaries. I know someone who does at his job, and even he only uses it for the boring, never acted upon meetings that everyone thinks is unnecessary but the managers think should be done anyways, because it just doesn’t work well enough to justify use on anything even remotely important.
Even just from a purely technical standpoint, the context windows of LLMs are so small relative to the scale of meetings, that they will almost never be able to summarize it in its entirety without repeating points, over-explaining some topics and under-explaining others because it doesn’t have enough external context to judge importance, etc.
But if you give it a single small paragraph from an article, it will probably summarize that small piece of information relatively well, and if you give it something already formatted like bullet points, it can usually combine points without losing much context, because it’s inherently summarizing a small, contextually isolated piece of information.
I think your manager has a skill issue if his output is being badly formatted like that. I’d tell him to include a formatting guideline in his prompt. It won’t solve his issues but I’ll gain some favor. Just gotta make it clear I’m no damn prompt engineer. lol
I didn’t think we should be using it at all, from a security standpoint. Let’s run potentially business critical information through the plagiarism machine that Microsoft has unrestricted access to. So I’m not going to attempt to help make it’s use better at all.
Hopefully if it’s trash enough, it’ll blow over once no one reasonable uses it.
Besides, the man’s derided by production operators and non-kool aid drinking salaried folk
He can keep it up. Lol
Nobody is a “prompt engineer”. There is no such job, for all practical purposes, and can’t be one given that the degenerative AI pushers change their models more often than healthy people change their underwear.
It really depends on the type and size of text you want it to summarize.
For instance, it’ll only give you a very, very simplistic overview of a large research paper that uses technical terms, but if you want to to compress down a bullet point list, or take one paragraph and turn it into some bullet points, it’ll usually do that without any issues.
Edit: I truly don’t understand why I’m getting downvoted for this. LLMs are actually relatively good at summarizing small, low-context-necessary pieces of information into bullet points. They’re quite literally made as code that interprets the likelihood of text based on an input. Giving it a small amount of text to rewrite or recontextualize is one of its best strengths. That’s why it was originally mostly implemented as a tool to reword small isolated sections in articles, emails, and papers, before the technology was improved.
It’s when they get to larger pieces of information, like meetings, books, wikipedia articles, etc, that they begin to break down, due to the nature of the technology itself. (context windows, lack of external resources that humans are able to integrate into their writing, but LLMs can’t fully incorporate on the same level)
But if the text you’re working on is small, you could just do it yourself. You don’t need an expensive guessing machine.
Like, if I built a rube-goldberg machine using twenty rubber ducks, a diesel engine, and a blender to tie my shoes, and it gets it right most of the time, that’s impressive. but also kind of a stupid waste, because I could’ve just tied them with my hands.
Even there it will hallucinate. Or it will get confused by some complicated sentences and reverse the conclusion.
It can, but I don’t see that happen often in most places I see it used, at least by the average person, although I will say I’ve deliberately insulated myself a bit from the very AI bro type of people who use it regularly throughout their day, and mostly interact with people who are using it occasionally during research for an assignment, rewriting part of their email, etc, so I recognize that my opinion here might just be influenced by the type of uses I personally see it used for.
In my experience, when it’s used to summarize, say, 4-6 sentences of text, in a general-audience readable text (i.e. not a research paper in a journal) that doesn’t explicitly rely on a high level of context from the rest of the text (e.g. a news article relies on information it doesn’t currently have, so a paragraph out of context would be bad, vs instructions on how to use a tool, which are general knowledge) then it seems to do pretty well, especially within the confines of an existing conversation about the topic where the intent and context has been established already.
For example, a couple months back, I was having a hard time understanding subnetting, but I decided to give it a shot, and by giving it a bit of context on what was tripping me up, it was successfully able to reword and re-explain the topic in such a way that I was able to better understand it, and could then continue researching it.
Broad topic that’s definitely in the training data + doesn’t rely on lots of extra context for the specific example = reasonably good output.
But again, I also don’t frequently interact with the kind of people that like having AI in everything, and am mostly just around very casual users that don’t use it for anything very high stakes or complex, and I’m quite sure that anything more than extremely simple summaries of basic information or very well-known topics would probably have a lot of hallucinations.
See, when I have 4-6 sentences to summarize, I don’t see the value-add of a machine doing the summarizing for me.
(Note: the above sentence is literally a summary of about a dozen sentences I wrote elsewhere that contained more details.)
Oh I completely understand, I don’t often see it as useful either. I’m just saying that a lot of people I see using LLMs occasionally are usually just shortening their own replies to things, converting a text based list of steps to a numbered list for readability, or just rewording a concept because the original writer didn’t word it in a way their brain could process well, etc.
Things that don’t necessarily require a huge amount of effort on their part, but still save them a little bit of time, which in my conversations with them, seems to prove valuable to them, even if it’s in a small way.
I feel like letting your skills in reading and communicating in writing atrophy is a poor choice. And skills do atrophy without use. I used to be able to read a book and write an essay critically analyzing it. If I tried to do that now, it would be a rough start.
I don’t think people are going to just up and forget how to write, but I do think they’ll get even worse at it if they don’t do it.
Our plant manager likes to use it to summarize meetings (Copilot). It in fact does not summarize to a bullet point list in any useful way. Breakes the notes into a headers for each topic then bullet points The header is a brief summary. The bullet points? The exact same summary but now broken by sentences as individual points. Truly stunning work. Even better with a “Please review the meeting transcript yourself as AI might not be 100% accurate” disclaimer.
Truely worthless.
That being said, I’ve a few vision systems using an “AI” to recognize product that doesn’t meet the pre taught pattern. It’s very good at this
This is precisely why I don’t think anybody should be using it for meeting summaries. I know someone who does at his job, and even he only uses it for the boring, never acted upon meetings that everyone thinks is unnecessary but the managers think should be done anyways, because it just doesn’t work well enough to justify use on anything even remotely important.
Even just from a purely technical standpoint, the context windows of LLMs are so small relative to the scale of meetings, that they will almost never be able to summarize it in its entirety without repeating points, over-explaining some topics and under-explaining others because it doesn’t have enough external context to judge importance, etc.
But if you give it a single small paragraph from an article, it will probably summarize that small piece of information relatively well, and if you give it something already formatted like bullet points, it can usually combine points without losing much context, because it’s inherently summarizing a small, contextually isolated piece of information.
I think your manager has a skill issue if his output is being badly formatted like that. I’d tell him to include a formatting guideline in his prompt. It won’t solve his issues but I’ll gain some favor. Just gotta make it clear I’m no damn prompt engineer. lol
I didn’t think we should be using it at all, from a security standpoint. Let’s run potentially business critical information through the plagiarism machine that Microsoft has unrestricted access to. So I’m not going to attempt to help make it’s use better at all. Hopefully if it’s trash enough, it’ll blow over once no one reasonable uses it. Besides, the man’s derided by production operators and non-kool aid drinking salaried folk He can keep it up. Lol
Okay, then self host an open model. Solves all of the problems you highlighted.
Nobody is a “prompt engineer”. There is no such job, for all practical purposes, and can’t be one given that the degenerative AI pushers change their models more often than healthy people change their underwear.
Right, I just don’t want him to think that, or he’d have me tailor the prompts for him and give him an opportunity to micromanage me.