Don't Miss the Boat on AI

Like most of the world, I have gotten swept up in the ChatGPT hype. I was using it a ton when it launched. Using it on personal projects, at work, and looking for ways to incorporate it into products. But as time went on the shine faded. It gave wrong answers, made things up, or I often felt I could do higher quality work without it. But the ChatGPT plug-ins have reignited my interest. I currently have access to web browsing and code interpreter plug-ins. I wanted to share a recent exercise I went through to test how plug-ins might impact how I work and build products.

DALL·E 2023-05-12 16.37.07 - A product manager, standing on the beach, missing the boat, .jpeg

Working with Plug-Ins

With the recent waves of layoffs, I was interested in future job trends for Product Managers. I asked ChatGPT to help build a data set on PM job trends. This proved to be difficult. Even with web browsing, it was unable to compile anything of use. Most of the needed data sources were paywalled or otherwise inaccessible. So if you want ChatGPT to tell you if product management is the job of the future, you are out of luck.

CleanShot Safari-2023-05-12.png

With PM jobs data being an issue, I turned my focus to something I knew would be accessible. I focused on a data set about Y-Combinator batch applicants and acceptances. ChatGPT did much better on this, likely because YC widely publishes stats on each batch class. After going back and forth a few times we had a data set ready for analysis. You’ll see that the data set is pretty basic. That is due to my impatience, not a lack of capabilities by ChatGPT. If I had wanted to, I could have kept directing it to build a more and more complicated data set.

CleanShot Safari-2023-05-12-2.png

One drawback of the plugin setup is that they work in isolation. Once I had the dataset ready, I had to start a new chat with the Code Interpreter. But besides that small amount of friction, the Code Interpreter is impressive. I uploaded the data set as a CSV, and ChatGPT got to work. It imported the needed packages, parsed the file, and provided a high-level analysis.

CleanShot Safari-2023-05-12-3.png

I asked it for some key takeaways and trends from the data. And then I asked that it generate some charts for the data. Based on the quality of analysis it did, I have confidence it could go into much more depth if data were available.

CleanShot Safari-2023-05-12-4.png

Lastly, I asked it to forecast the number of applicants that would be accepted over the next 3 years. I told it to consider the variance in Winter and Summer batches. It explained to me the possible approaches for doing this type of forecast. It also called out the limitations we were up against due to the data we had. And then it outlined the approach it recommended we should take. Its recommendation matched what I would have used if I had done this on my own. This gave me some confidence that it knew what it was talking about (or it could mean I have no idea what I am doing). Once it generated the forecast it gave me an explanation of the results. I then asked it to plot the results on the same chart as the historical data.

CleanShot Safari-2023-05-12-5.png

This is where things got squirrelly, no matter what I asked it at this point it would crash. I tried a bunch of different things but they all ended with it spinning and crashing. I'm still not sure if it was related to me hitting a usage limit or some other issue. But after a few minutes, I gave up. I tried picking it up again later that day and even the next day. But every time it crashed. Even with things ending as they did, I got enough out of the exercise to understand the value.

The Good

There is quite a bit to like about the plug-ins:

CleanShot Safari-2023-05-12-6.png

The Bad

Not everything was great though, a few things bugged me or caused friction:


Specific to what I tried here there were two things that stuck out. The first was how well it did a ton of work for me. I am capable of gathering data, cleaning data, analyzing it, and visualizing it (this part is my actual job). But it was nice to be able to pass that off to someone else. Its ability to do high-quality work across these domains, with little instruction, is impressive. (This is especially true for a product still in Alpha.) The other big thing to like is that it teaches you as it goes along. Not only does it do the work for you and provide analysis of the data, but it also tells you what it is doing and why it is doing it. This is helpful in cases where you are new to an area. You can learn methods and techniques that you can adopt in your own work.

Outside of this particular exercise, here are some broader takeaways on how tools like these will impact products. 

I’m still interested in AI, in all its forms, including ChatGPT. But despite the potential of these plug-ins, I am still cautious. Cautious that AI is the answer to all the problems we face in product and tech. As was always the case it is on us to do the work, solve the right problem, in the right way. Pick the tool that is best for the job. Don't pick AI chat just because everyone else seems to be doing so.