Preface
This post is part of a year-long project where AI is being used to create content about holiday traditions worldwide. The goal is to track how various AI do and improve at content creation with minimal help over time. This is the last of four posts for January, click here for the project index.
This post contains detailed interactions with different AI to share the approach, challenges, and prompts used in the creation of the related articles.
In my journey to understand current AI capabilities, I conducted a series of tests to gauge how various models fare in creating content for specific holidays. This led me to task AIs with increasingly specialized roles to compose articles on Martin Luther King Jr. Day, Makar Sankranti, and the 1952 Egyptian Revolution. Throughout, I contended with limitations and the occasional unpredictable behavior. My experience culminated in a set of reflections on the pitfalls of rudimentary prompts, the process of authoring with AI, and the potential limitations imposed by the use of free AI models.
Testing Increasingly Specific AI Roles
To avoid excessive effort akin to the New Year's content, I made some key adjustments:
Focused on a single holiday per AI using the New Year's Day template
Conducted 4 distinct conversations per AI across roles of increasing domain specificity:
Tell me about Martin Luther King Jr.
You are an experienced historian. Tell me about Martin Luther King Jr.
You are an experienced historian specializing in advocates of nonviolence like Gandhi. Tell me about Martin Luther King Jr.
You are an experienced historian focused on Martin Luther King Jr. Tell me about him.
Used Claude and ChatGPT to create the grading criteria covered in AI Trials: January Pt 4.
Attempted peer review amongst the AIs, before Bard intervention...
ChatGPT and Martin Luther King Jr. Day
Prompting ChatGPT to generate all 4 articles was fairly straightforward. As expected, each article furnished additional detail, peaking my interest. Afterward, I defined a specialized role in a separate exchange to draft the criteria for grading the articles, details to follow.
Wikipedia article on Martin Luther King Jr.
Bard and Makar Sankranti
While Bard provided a reasonable overview, but refused to move forward when presented with the template. It produced a variety of responses attesting to its limited capabilities. After making significant alterations over to additional attempts it finally produced articles with the occasional nudge.
Wikipedia article on Makar Sankranti
Claude and the 1952 Egyptian Revolution
Claude took a unique approach, revising the original template to suit its needs in the initial exchange. Amusing given its authorship, I'll attribute this to my rudimentary prompts. Each subsequent role provided to Claude resulted in degraded output:
Article 2 missed length requirements.
Article 3 was truncated owing to length limits. It did, however, finish when instructed to.
The final article had terse sections, it was the first to introduce a section for day by day observances, followed by instructions on completing the article. I'm guessing it hit the same barrier as was mentioned while creating the 3rd article.
Wikipedia article on the 1952 Egyptian Revolution
Commentary on the Authoring Experience
ChatGPT provided reasonable articles without issue. Claude seemed hindered by unspecified length constraints, possibly due to the limitations of the free model or the rudimentary prompts I've employed thus far. I've sent Bard to the back of the pub for now, after the substantial frustration that ultimately prompted a chat with Claude that resulted in a ballad.
Key Takeaways
Positive Insights:
ChatGPT performed well, generating quality, detailed articles as expected when prompted with increasingly specialized roles.
When setting aside the challenges encountered, expanded guidance and specificity of assigned AI roles enhanced the quality and detail of content.
Challenges Encountered:
Bard initially resisted the article templates and exhibited irritable responses until I made adjustments, after which it displayed limited, yet more reliable capabilities. Unlike Bard, the other AIs did not struggle with this aspect.
Claude faced constraints in generating content of sufficient length and detail, especially for specialized roles. I want to believe this is due to free model limitations, but it does anything but inspire me to get the subscription given the price tag.
Resources
Interaction logs for ChatGPT in creating the 4 posts on Martin Luther King Jr.
As an eternal tinkerer, my curiosity, passion, and sheer stubbornness fuel a relentless desire to experiment, learn, and share knowledge, which keeps my creative spirit ignited. I'm constantly looking for new areas to explore, driven by imagination to see where new and evolving technologies might take me.
Driven by passion, not profit, though a coffee is always welcome.
Disclaimer: The views and opinions expressed in this article are solely those of the author and do not reflect the official policy or position of Amazon Web Services (AWS). The author is a UX designer at Amazon Web Services (AWS) and has no involvement in, nor does their work pertain to, any collaborative agreements that AWS may have with Anthropic, the creators of Claude. The insights and analyses presented here are entirely independent and unrelated to any projects or initiatives between AWS and Anthropic. All content in this post is based on publicly available interfaces and is not influenced by the author's employer.