Welcome back to my series on using ChatGPT for programming (written with help of ChatGPT)! In this second installment, I’ll share my experiences using ChatGPT to build a small software project.
Here you can find the chats: https://github.com/juangamnik/chatgpt-schedule/tree/main/chatgpt
I set out with the premise that I would not write more than a few lines of code, leaving most of the work to ChatGPT. Let’s dive into the numbers, the areas where ChatGPT slowed me down, and the moments where it truly shined.
A Closer Look at the Numbers
While working with ChatGPT on this small software project, I observed some interesting statistics that provide insight into the process. Building the project manually would have taken me around 4 hours, but with ChatGPT’s assistance, it took 16 hours, making the process four times slower. Despite the increased time, ChatGPT was helpful in some areas, like accelerating the writing of the README.MD file and retrieving the API_KEY for Google Calendar.
However, it’s important to note that there was no build pipeline and no tests during the project development. The final program consisted of 316 lines of code, with 164 lines in JSON, 118 lines in TypeScript, and 34 lines in Markdown. Some of the JSON code came from npm and Visual Studio Code, while I wrote a few lines of the Markdown code myself.
Throughout the project, I used 78 prompts and wrote 350 lines within those prompts. ChatGPT generated a total of 3039 lines of text for me to read and analyze. The numbers reveal the extent of interaction with ChatGPT and the considerable amount of guidance needed to complete the project.
Why Did ChatGPT Slow Me Down?
While working on this project, I encountered several challenges with ChatGPT that slowed down my progress. One major issue was with iterative programming. When I had to make changes to a function signature, it led to additional prompts in dependent chat threads and a significant amount of copying and pasting. This process could be much more efficient with a dedicated coding copilot.
Another aspect I noticed was ChatGPT’s eagerness to please. It reminded me of a Labrador dog and its “will to please”, that keeps it trying even when given an impossible task. For example, ChatGPT attempted to wait for an async call in a non-async method, which is only possible in top-level functions (see 03-get-holiday-dates.md). This resulted in an ineffective solution.
ChatGPT also seemed to have a superficial understanding of certain concepts, making it resemble a non-reflecting junior developer. One instance was with the API call eachWeekendOfInterval(), which ChatGPT used incorrectly. It tried to call two parameters instead of one and struggled to find a solution based on the compiler’s hints. I eventually had to consult the documentation and provide specific guidance to solve the issue (see 02-get-sundays.md).
Lastly, I faced challenges with time zones (UTC vs. UTC+1) and the output in the iCal format (see 08-resolve-timezone-issue.md). This issue added to the slow progress while working with ChatGPT. It’s interesting to see that ChatGPT faces the same issues as human programmers, such as calendar-related problems, character encoding issues, null pointers, and time zone discrepancies. Since ChatGPT has been trained on code written by humans, it’s not surprising that it shares these struggles. This experience highlights the potential for improvement in both AI-driven tools and human developers as they tackle common programming challenges.
These factors combined led to a slower development process when compared to building the project manually.
Where Did ChatGPT Shine?
Despite some of the challenges I faced while working with ChatGPT, there were instances where it truly shined and demonstrated its potential as a valuable programming assistant.
One such moment was when I encountered an issue with the index.ts file of the program. The call started with ts-node index.ts, causing the first parameter to the program to be process.argv[2] instead of process.argv[1]. When I provided ChatGPT with this information, it suggested using the minimist library to address the problem, rather than simply shifting the parameters by one. This was a far superior and more robust solution that showcased ChatGPT’s ability to think beyond basic fixes.
Another impressive moment came when I was searching for a bug in ChatGPT’s call to the Google Calendar API. I asked ChatGPT to provide me with logging code to pinpoint the issue. After analyzing the results of the logging code, ChatGPT was able to identify the exact two problematic lines and fix the problem (see 03-get-holiday-dates.md). This demonstrated ChatGPT’s capability to effectively diagnose and resolve issues when given the right context and guidance.
These moments were shining examples of ChatGPT’s potential and highlighted the benefits it can bring to a programming project when leveraged correctly.
Conclusion
Reflecting on my second attempt at using ChatGPT for programming, the results were a mixed bag. While it took me four times longer to complete the project with ChatGPT, there were instances where the AI proved its worth. For example, it accelerated writing the README.MD file and retrieving the API_KEY for Google Calendar. It also showcased its potential in areas like debugging, and suggesting more robust solutions when given the right context.
However, this experience also revealed several limitations of ChatGPT. Its tendency to attempt impossible tasks, as well as its occasional shallow understanding in certain areas, can be problematic. These issues highlight the importance of understanding and mitigating the limitations of AI-driven tools like ChatGPT when using them in a programming context.
Despite these challenges, I remain optimistic about the potential of ChatGPT and similar AI-driven tools in the software development process. With further development, refinement, and a better understanding of their strengths and weaknesses, these tools could become valuable assets for programmers. As we learn from these experiences and work to improve the technology, I eagerly anticipate a future where AI companions can effectively assist us in building software projects, enhancing our productivity and creativity along the way.
Amendment
It is amazing to see the fast pace of development in recent AI solutions like ChatGPT. In the first iteration of this software project, which I described in the previous blog post, I asked ChatGPT about calculating holidays like Easter. At that time, the AI struggled to provide a satisfactory solution. However, in a later iteration, ChatGPT successfully offered a working solution using the Meeus/Jones/Butcher algorithm.
While it’s impressive to see the AI’s ability to adapt and improve, it’s important to note that a seasoned developer would likely have suggested using an existing library for this task. This highlights an area where AI solutions like ChatGPT still have room for growth. By learning to leverage existing resources and recommending the most efficient solutions, AI-driven tools can become even more valuable assets to developers.
The speed at which AI technology is advancing is truly amazing, and this experience demonstrates the potential of AI-driven tools in the software development process. As these tools continue to evolve and refine their capabilities, it’s exciting to envision the countless possibilities they will bring to the world of software engineering.
Exciting 🤓.