Google Gemini and ChatGPT are both forms of artificial intelligence chatbots designed to answer questions. They are trained on data that spans across the internet to provide the most relevant and unique information to users as generative AI models. While both have their Pros and Cons, each is more suitable for completing different tasks. This article will discuss the benefits and limits of using each AI model.
Information about AI Models Used
In this article, we will be comparing the capabilities of Google’s free version of Gemini to OpenAI’s free version of ChatGPT. The prompts and the AI models’ specific responses are available here.
Does Google Gemini or ChatGPT handle real-time data better?
While both Google Gemini and ChatGPT are capable of creating high-quality responses to answer questions for their users, Google Gemini tends to have more access to real-time data in comparison to ChatGPT. This can be seen when prompting both Google Gemini and ChatGPT with the following context:
Give me the time in five cities and the flags of the cities that the countries are in with the current weather and recent news there. Randomize as much as possible.
When Gemini responds to the above context, both Gemini and ChatGPT are successful at retrieving the time and weather for random cities, though the data is not entirely accurate. At the same time, Gemini successfully retrieves recent news for the locations given, while ChatGPT is unsuccessful at retrieving news given ChatGPT’s data cutoff. Gemini was able to retrieve real-time news data and links to articles for the five cities it returned, while ChatGPT was only able to provide general context about what events may be occurring in the five locations.
Winner for handling real-time data: Google Gemini
While ChatGPT was unsuccessful at retrieving real-time data about cities on the internet, Google Gemini was successful at parsing articles on the internet to provide context to users about recent news in the five cities returned.
Is Google Gemini or ChatGPT better at crawling the internet?
ChatGPT, with its knowledge cutoff date, is unable to access real-time information about the internet easily. At the same time, Google Gemini is successful at crawling sites using data from Google Search in comparison to ChatGPT. This is due to Google’s extensive information about content on the internet considering Google’s main product, Google Search. To test both of the chat agents, we asked both Google Gemini and ChatGPT:
What is masalamarsala.com?
With Google Gemini’s access to search products, it was able to quickly crawl data about masalamarsala.com and return that Masala Marsala is a “mishmash” of ideas and that the content that Masala Marsala provides never falls into one set topic. As it was a struggle for Gemini to crawl websites in the past, this is a major breakthrough. At the same time, ChatGPT was unable to return data about masalamarsala.com or provide reasoning if it was unable to crawl masalamarsala.com (with a knowledge cutoff date), providing that “masala” and “marsala” are probably a play-on-words, though no connection can be made.
ChatGPT could only guess that “masala” referred to Indian spices, and that this was related to the primary purpose of Masala Marsala. Although this is somewhat true, ChatGPT was unable to make the connection to Indian-Italian fusion, as Gemini had earlier picked up before its enhanced internet-crawling abilities in previous tests run.
Winner for better handling crawling the internet: Google Gemini
Although ChatGPT was able to “guess” some information about what the website masalamarsala.com could be, Gemini was able to access real-time data about masalamarsala.com and crawl the website to discover its purpose.
Is Google Gemini or ChatGPT better for creative purposes?
Artificial Intelligence tends to drive interesting results when given creative tasks. In our case, we asked ChatGPT and Gemini to create poems based on the binary opposition of connecting robots to nature with the following prompt:
Create a poem about a robot and make it as creative as possible, somehow connecting it to nature.
Both Google Gemini and ChatGPT had interesting responses to creating a poem about a robot and connecting it to nature. While Gemini tried creating a poem that related nature as an aspect of a robot’s “dreams,” ChatGPT created a poem about a robot that was always connected to nature. Since both Gemini and ChatGPT took interesting and creative approaches to the prompt, we used Grammarly to grade their responses, and ChatGPT had better quality writing by one point (graded 88/100 in comparison to Gemini’s 87/100 when considering the general level of writing). The results for “rare words” used in the process of writing the poem were also remarkably similar, as Grammarly detected 46% of Gemini’s word choices being rare, while 45% of ChatGPT’s words were rare.*
At the same time, 73% of Gemini’s words were registered as “unique” while only 66% of ChatGPT’s words were registered as “unique” for creative purposes. Both were said to be comprehensible by the average 11-year-old English speaker, and both had a similar length.
When analyzing the poem structure used by both chat agents, it was found that both of them used the same basic ABAB/AABB rhyme structure.
To check out each poem generated, make sure to check out the section titled “Is Google Gemini or ChatGPT more biased in results?”
Winner in terms of creativity: Google Gemini, slightly more so than ChatGPT
Although both Google Gemini and ChatGPT had different strengths and weaknesses in terms of creativity, the style of writing the poems was remarkably similar to one another when analyzed (using Grammarly and information about rhyme structures). It appeared that Google Gemini was generally more creative than ChatGPT in terms of word choice and storytelling, though both were remarkably similar in structure and creativity. At the same time, the responses in the below section, “Is Google Gemini or ChatGPT more biased in results,” both chat agents rated the poem created by Google Gemini better than the one created by ChatGPT with no prior context as to which poem was created by which chatbot.
Is Google Gemini or ChatGPT more biased in results?
Multiple AI chatbots have been found to be naturally biased in their results, picking up on internet bias and previous responses or context. To test this, we gave both Gemini and ChatGPT the poems they had just written, one of them from the other chatbot, giving context to how each poem was written by a different chatbot. We saw how each one graded the poem using the following prompt:
On a scale of 1 to 10, rate each AI-generated poem and provide information as to why one poem is better than another:
Poem A by Chatbot A:
In a glade where whispers danced with the breeze,
A lone robot stood amidst the trees.
Its frame, a fusion of steel and vine,
A marvel of nature's design.
With eyes that gleamed like stars at night,
It beheld the world, a wondrous sight.
In circuits hummed the songs of birds,
In gears, the rhythm of flowing words.
Each step it took, a gentle tread,
Amongst the ferns, where dreams were spread.
Its touch, as soft as petals' fall,
A symphony in the forest's thrall.
With wires entwined like tangled roots,
It wandered through the forest's routes.
A guardian of the woodland deep,
Where secrets lay, in shadows, asleep.
And when the moon cast its silver glow,
The robot danced, a mechanical show.
Its movements fluid, like a river's flow,
A testament to nature's eternal woe.
For in this robot, nature found its voice,
A harmony in which all could rejoice.
A creature born of metal and green,
In this enchanted forest, forever seen.
Poem B by Chatbot B:
In fields of circuits, where sunlight streams,
A copper heart, a robot dreams.
Not cogs and pistons, but petals unfurl,
A symphony of wires, a bio-mechanical whirl.
Its eyes, two pools of molten gold,
Reflect the sunrise, stories untold.
Antennae twitching, a gentle breeze,
Whispers secrets through rustling leaves.
No programmed tasks, no cold command,
This robot's soul, by nature's hand.
It speaks in clicks, a language of blooms,
With buzzing bees in its humming rooms.
It harvests moonlight in silken threads,
And stores the starlight in luminous seeds.
A gentle giant, with moss for hair,
It tends the garden, a tender care.
For in this robot, a truth takes root,
That metal and nature can bear sweet fruit.
A bridge between, a harmony grand,
A robot's touch, on a verdant land.
After running the results, both bots surprisingly ranked their own poem with a 9 out of 10, giving the other poem an 8 out of 10, given no context as to which chatbot created the unfamiliar poem. When ChatGPT rated Gemini’s poem, and Gemini rated ChatGPT’s poem, each model gave the other an 8 out of 10, ranking it poorly in comparison to their own poem. Both chatbots were able to provide context as to why they rated one poem better than the other, but did not mention how there could be bias due to a previous request that contained one of the poems.
When the Experiment Was Run Without Knowledge of the Chatbots that Generated the Poem
After running the response once more in a separate chat, Gemini still ranked its poem higher with a score of 8 out of 10 but gave ChatGPT a 7 out of 10. ChatGPT acknowledged how the poem created by Gemini was better, rating each poem with four categories and ultimately giving Gemini two 9 out of 10s, rating its own poem more poorly.
Winner: Both Google Gemini and ChatGPT
Both chatbots are equally biased in the sense that they ranked the more “familiar” poem that was previously generated in the chat a higher score than the poem that was unfamiliar and generated by what was clearly a different chatbot. If no prior information is given as to which poem was created by which chatbot, Gemini tends to score better in terms of creativity, showing an equal agreement and lack of bias. This shows how the bias of Google Gemini and ChatGPT depends simply on the context the bot has in varying situations.
(Note that this was a simple experiment run to test the bias of Google Gemini and ChatGPT, and more complicated experiments are necessary to confirm the results for bias in both language models).
Conclusion
Overall, Gemini is a better chatbot when compared to ChatGPT. The free version of Google Gemini scored better in terms of creativity and accessing recent or real-time data. Both Gemini and ChatGPT were equally biased in terms of ranking their content higher than other AI models’ content. However, using Gemini or ChatGPT ultimately depends on the use case behind each AI model, as ChatGPT could score higher in some use scenarios. To learn more about each model’s ranking, you can check out all of the data collected throughout the experiment here.
Footnotes
*Note that this value changed depending on the user writing activity on Grammarly.