Microsoft 365 Copilot, a generative AI integrated into Microsoft applications, promises a wide range of functions across the MS Office application suite. Depending on the Office application, Copilot can create documents and presentations, summarize meeting notes, perform analytics, and much more with a single click – at least if you believe Microsoft’s claims.
During the first Copilot presentation, Microsoft showed some impressive use cases. But how well do they work in the real world? We at empower® put Microsoft 365 Copilot to the test and we’d like to share our findings with you. In this article, we will focus on Copilot in Excel and Word.
The test framework
Ten empower employees spent a month testing Microsoft 365 Copilot. They tested numerous features, both according to the use cases communicated by Microsoft and on the basis of company-specific requirements.
In this report, we present the test results for three applications: Word, PowerPoint and Excel. Each task was accomplished by typing an appropriate command into a chat window, also known as a prompt. You’ll see how we formulated the prompts in detail and how well or poorly Copilot responded to them.
First, an important note: for this test, we used the first released version of Copilot from November 2023. This is important because we can expect regular updates in the future, which should improve Copilot continuously.
Copilot in Excel
Excel poses significant challenges for many users, especially those who don’t work with the application on a regular basis. This makes Copilot's potential in this context all the greater. In the version we tested, Copilot for Excel was still in preview mode.
After entering our first query, Copilot told us that the data had to be in tabular form for it to understand. So data must first be formatted before Copilot can work with it.
1. Data analysis and knowledge gain
“Highlight the highest values in ‘Total Invoice (in €)’”
Copilot easily implemented a request to highlight a specific value in the table. In the chat area, Copilot also showed which cell was highlighted in color. This is especially helpful for large and complex tables.
We tested other data queries as well:
“Bold the top 10 values in ‘Pro Rata (Gross)’”
“Show only non-project expenses”
Copilot easily handled simple analysis and sorting tasks. This is especially useful for people who don’t use Excel regularly and have only basic Excel skills.
2. Data aggregation
We wanted to learn more about our data and asked Copilot for insights. The results were first displayed in the chat bar and could be pasted into a new worksheet using a prompt.
“Insights for net amount”
“Can I see another insight?”
“Add all insights to grid”
The results provided a good overview of the data. Copilot is especially helpful for this task if you have only basic Excel skills or aren’t a data analyst.
3. Complex Excel models
We wanted to see how Copilot would perform with complex models and analyses, such as those used in investment banking. A valuation model in Excel served as the basis for this use case.
“Make a graphical overview of the most important figures (both valuation and operational)”
Copilot gave us country-specific results, which was acceptable because the prompt left a lot of room for interpretation. Again, the result was displayed in the chat bar and could be transferred to a new worksheet with a single click. Copilot provided a clear summary of the most important data.
“Add to a new sheet”
Copilot easily added the results to a new sheet.
However, when asked for more complex analyses, in our case for a specific valuation range, Copilot couldn’t provide answers.
“What would be a valuation range (Enterprise value) for a comparable company that has a sales growth of 10% annually until 2025, an EBITDA of €100m with a margin of 50%?”
“How can the multiples be best explained given the operational statistics?”
Copilot didn’t provide an appropriate result for this task either. Instead, the AI gave us the same result as in the first use case.
Finally, we asked Copilot for statistics for certain KPIs. Here, the AI couldn’t find any matches, couldn’t calculate the data from the table, and therefore couldn’t present a result. To be fair, this last use case was really complex and difficult.
“Create operational statistics: Sales, EBITDA, CAPEX | Sales growth, EBIT growth, EBITDA growth | EBIT margin T, EBIT margin T+1, EBIDTA margin T, EBIDTA margin T+1”
Our conclusion about Copilot in Excel is that for those who aren’t Excel professionals or don’t use the application on a daily basis, Copilot can certainly provide some assistance. The AI delivers a quick overview of data and increases efficiency when working with Excel.
Copilot currently seems less useful for advanced uses because Copilot can’t yet perform complex analyses. The restriction that all data must be in tabular form requires an extra step in practice, but thanks to the formatting options, that can be done with just a few clicks.
Note that we were only able to communicate with Copilot in Excel in English. In all other applications, it was possible to write prompts in German, which may be due to the preview mode.
Copilot in Word
We took a closer look at Copilot in Word. Our expectations were very high, as ChatGPT is quite powerful in text creation and editing.
1. Create a Word document based on a given presentation
“Create a proposal for a customer using Brand Control EN.pptx. Add a table comparing the features of Brand Control and Microsoft Copilot using Microsoft-copilot-whitepaper-empower-en.docx. Make the document look like Brand Compliance in Microsoft Office Proposal.docx”
We provided Copilot with a PowerPoint presentation and a Word document to create a document based on the presentation.
The content of the document Copilot produced is very good and detailed. The information was presented correctly and in a structured way, including (sub)headings, bullets, and tables. Copilot followed the presentation closely. In addition, Copilot has taken other content from the Internet, but marked it accordingly.
Unfortunately, the transfer of the design we provided as a Word document didn’t work. Despite the impressive results in terms of content and time saved, its practical application is limited due to the extra formatting required.
2. Formatting a document with style sheets
Since Copilot hadn’t adopted the design in the previous use case, we took it a step further. We gave Copilot specific guidelines regarding our corporate design that it should use to format the Word document. Those guidelines included information about fonts and font sizes for the headings.
“Format the document using the heading and text styles defined in this Word template C:\Users\mvsmq\OneDriveDesktop\BasicBrandGuidelines.docx”
Instead of implementing the instruction, Copilot pointed us to the Effects feature of Microsoft Word. As we discovered with PowerPoint, Copilot doesn’t yet seem able to format documents or presentations in a brand-compliant manner.
3. Company specific spellings
Almost every company uses certain customized spellings, especially for products and services. At empower®, for example, this includes the ® symbol after the product name “empower”. We instructed Copilot to take this into account.
“Always write the word empower with a ‘®’, like this: empower®”
However, our test showed that Copilot couldn’t use the correct spelling in the document. It suggested that we try a different formulation of the prompt. Which we did:
“Change the word empower or Empower in the document like this: empower®”
But even the customized prompt didn’t produce the desired result. Copilot gave us a hint on how to proceed and pointed us to the “Replace” feature. That was actually helpful in this specific example, even though our overall expectations were higher. Unfortunately, this isn’t useful in every use case.
The results of this test surprised us because ChatGPT, on which Copilot is based, normally handles such requests.
4. Rewrite or shorten a text
In the fourth use case, we asked Copilot to paraphrase a text. This doesn’t require a prompt, as Copilot offers a paraphrase function automatically.
For paraphrasing, the interface is very intuitive and user-friendly. It allows you to completely replace your own text with the rewritten version, generate a new text, or insert the new text under the old one. The clear display of the texts one below the other ensures easy review. This corresponds to the comfort and quality we expected from ChatGPT.
Overall, however, our Copilot didn’t meet our expectations in Word. While simple tasks such as creating or rephrasing texts worked well, more specific formatting and wording requirements failed.
Our conclusion
Microsoft 365 Copilot is especially helpful when creating or editing text. Whether it’s researching, summarizing, or rewriting, Copilot handles these tasks very well.
However, we have yet to see it revolutionize everyday office life. Many of the use cases presented by Microsoft and their results didn’t pass our real world tests. When it comes to more complex queries, the AI still too often reaches its limits. The same goes for adherence to brand specifications.
When it comes to Excel, Copilot is helpful for people with basic skills. However, the AI doesn’t yet offer significant benefits for Excel professionals.
In Word, similar to Outlook and Teams, Copilot can assist by paraphrasing, summarizing, or giving text a different style.
Working with Copilot has been a pleasure and has given us an interesting taste of what is to come. The potential is considerable and we’re excited to see what improvements will come with the next updates from Microsoft.
Would you like to learn more about Microsoft 365 Copilot in other applications? We have also tested Copilot in PowerPoint, Outlook and Teams.
Do you already know the empower® AI Assistant? It can be easily integrated into Microsoft 365 applications and makes your day-to-day work easier. For example, you can easily optimize your PowerPoint presentation or create a Word document in a matter of seconds.
The best time management tip: Keep your inbox empty!
Global rebranding strategy in action
