This is the first blog post in a row that describes my first experiences with ChatGPT as a pair programmer and assistant for a developer. As you will see, these experiences had some ups and some downs, but all in all – spoiler alert – on the one hand is the advancement in A[G]I (Artificial [General] Intelligence) especially in regard of NLP (Natural Language Processing) using LLM (Large Language Models) and GPT (Generative Pre-trained Transformers) very impressive. On the other hand I had to revise some of the observations (and criticism) I made, just weeks later, since the pace of evolution in the AI scene is so freaking high.
So these are the blog posts of this series:
- This post
- The ChatGPT Programming Experience: A Second Attempt
- The Evolution of Large Language Models in Programming: A Broader Perspective
But let’s start at the beginning (the following text has been created with help of GPT-4)…
Introduction
As a developer and software architect with extensive experience in test-based modeling and model-driven software engineering, I was both intrigued and skeptical when I first saw videos demonstrating the capabilities of ChatGPT and Codex. This AI model seemed to effortlessly create complete programs and even games, so I decided to put it to the test myself. In this first installment of a series of blog posts, I’ll detail my initial experience with ChatGPT, the challenges I faced, and my decision to give it a second chance.
First Try: A Typescript Program for Creating an Event Plan
On December 31st, 2022, I set out to create a Typescript program that generates a randomly distributed schedule for a given set of names of individuals for each Sunday of the year, as well as for Christian holidays with divine services in Germany. The goal was to write the schedule to a file in the iCAL format, allowing for easy import into Google Calendar. I gave me (and ChatGPT) one evening to complete this task.
The First Hurdle: Calculating Easter and Dependent Holidays
My initial experience with ChatGPT was disappointing. The AI provided vague answers and incomplete code, expecting me to fill in the gaps. When I insisted it to calculate Easter and dependent holidays, ChatGPT attempted the task using a completely incorrect algorithm. Eventually, I convinced it to use a library, but it requested that I add an extension function to the library instead. Frustrated, I added the necessary code using the library myself.
The Second Hurdle: Outputting the Schedule in iCal Format
The generated iCAL file appeared correct at first glance, but Google Calendar failed to load it. I used an online validator to identify the issue and tried to guide ChatGPT in fixing the problem. After two hours of fruitless back-and-forth, I capitulated and corrected the code myself.
A Second Chance for ChatGPT
Following discussions with friends and colleagues, I decided to give ChatGPT another chance. For my second attempt, I resolved not to write more than a few lines of code, requiring ChatGPT to rely heavily on libraries to address the shortcomings in solving complex problems, I observed in my first try.
In the next blog post, I’ll describe my further experiences with ChatGPT, conducted under more controlled conditions. I’ll share concrete numbers, anecdotes, and a GitHub project with all the chat logs and the resulting functioning software. Besides assisting with software development, ChatGPT also helped me with documentation and tasks like retrieving an API key for Google Calendar (and writing these blog posts).
Stay tuned for the next installment, where I’ll delve into the details of my second attempt with ChatGPT, exploring its potential and limitations as a software development tool.
Exciting 🤓.