Riding the Hype With ChatGPT for Business: Our Results Creating Audiences from Natural Language Requests

This blog post on ChatGPT for business is the last of a series we started on artificial intelligence (AI) and Generative AI, with opportunities in GenAI, AI challenges and how it could help with the creation of audiences as one of many potential AI use cases.

There is no doubt that the wave of technology revolutionized the world — and marketing — in the past 20+ years. The next era? It’s certainly going to be led by AI.

Generative AI — providing the ability to generate text, images or other media from prompts — received an accelerated awareness via ChatGPT tool from OpenAI. But is it “ready”?

What is ChatGPT?

Created by OpenAI using a neural network trained on a massive web scraped corpus, ChatGPT is a Large Language Model (LLM) with billions of parameters. ChatGPT allows users to submit natural language prompts and get back a range of textual responses including code, answers, generated text and more.

ChatGPT for Business as a Copilot

Readiness could mean different things to different people. Here, we are not looking to validate whether ChatGPT could replace business users to make customers happy and generate companies revenue.

Instead, we are looking to see if ChatGPT — along with AI in general — can become a copilot to reduce the friction between business users and technology.

Among the many places where this can be done, we decided to focus on one task that every ActionIQ user utilizes our platform for: creating audience segments.

The challenge for ChatGPT: As a business user, I want to create an audience using my own language.

From there, the business user should be able to take any action he would normally take such as getting insights from the audience, orchestrating and activating it.

The most complex and interesting part is really to evaluate the translation of a natural language request into SQL, which has been our focus in this research.

How Does ChatGPT for Business Work?

Several models are already available and trained on code. Some — reliably within certain complexity constraints — can generate SQL to query data based on a natural language description.

Step 1 – Prepare and submit a complete prompt

Our model is fed with a prompt structured in the following way: [ActionIQ Context] + [User Prompt]. The combined prompt will be sent to the model.

ActionIQ Context

While the use of ChatGPT interface allows end users to build a conversation where the previous prompts and answers are remembered and leveraged, the API has no “memory.” It means that the entire context for the request needs to be included in one block. With ChatGPT 4, the current prompt limit is 4096 characters, requiring us to carefully select what is being sent along with the user prompt.

The following three types of informations are going to be shared with our model:

1. SQL schema definition
2. Join column definition
3. SQL queries example

1. SQL schema definition

To inform the table structure in the database.

CREATE TABLE IF NOT EXISTS products (
	product_name LONGTEXT NULL,
	sale_price DOUBLE NULL,
	product_id INTEGER NULL, …
CREATE TABLE IF NOT EXISTS tc_leads (
	consumer_uuid LONGTEXT NULL,
	tc_v_cookie LONGTEXT NULL, ….

2. Join columns definition

To inform how tables can be linked to each other

customer_master.guid = tc_clickstream_events.guid
customer_master.guid = tc_leads.CONSUMER_UUID
customer_master.guid = tc_sales.CONSUMER_UUID
emails.email_aiq = customer_master.email_aiq
products.product_id = orders.product_id

3. SQL queries examples

To understand the output type we are looking for

SELECT (SUM(t_1.sale_amount)) as Net_Revenue FROM customer_master t_0 LEFT JOIN orders t_1 ON (t_0.customer_id = t_1.customer_id) WHERE (TRUE)
SELECT (CASE WHEN t_0.email_is_valid = 1 THEN true ELSE false END) as Email_Marketable FROM customer_master t_0 WHERE (TRUE)
…and a few more queries…

User Prompt

This is simply adding the prompt typed by the user into the overall request for the model

Step 2 – Clean Up Result

Finally, we are parsing the reply and taking some additional actions prior to surfacing the result to the user.

ChatGPT for Business Performance: It Can Do Better

Model code-davinci-002 discontinued for a reason

We first tested the SQL generation with code-davinci-002 model, and let’s say that we lost some hair getting very unpredictable results (hallucinations going to the roof).

Request:

write one SQL query surrounded by `<<` and `>>`
that returns unique customer_id for all customers with first name
equal to 'john'

Response:

HttpResponse({"id":"cmpl-6uoRs25zAZwBgunQQWF7LE8RyW6Ho","object":"text_completion","created":1678998312,"model":"code-davinci-002","choices":[{"text":"SELECT t_0.customer_id as Customer_ID <<customer_master.customer_id>> FROM customer_master t_0 as customer_master oliasdfkalsdfjalsdfaldsfjaslfjasdfjasdfusdfsl;fsdfl'^FDFL;F\n\nWHERE (t_0.firstname = 'mitesh') LIMIT 101\n\n### ORDER BY \nSELECT t_0.customer_id FROM customer_master t_0 as 'customer_id'; WHERE (t_0.firstname = 'john') LIMIT 101 ORDER BY t_0.customer_id ASC\n]]\n\nlocal ids = list_to_set{\n  29,<<'customer_master'>>,<< 'emails'>>,<<'customer_master.email_aiq'>>,<<'emails.email_aiq'>>,<< 'customer_id'>>,<< 'customer_master.customer_id'>>,<< 'emails.customer_id'>>,30,68,71,<<'orders'>>,<< 'orders.customer_id'>>,75,31,<<'products'>>,<< 'orders.product_id'>>,<< 'products.product_id'>>,32,35,67,<<'tc_clickstream'>>,<< 'tc_clickstream.customer_id'>>,80,<<'tc_leads'>>,81,<< 'tc_leads.customer_id'>>,84,86,<<'tc_sales'>>,<< 'tc_sales.customer_id'>>,91}\nl

Just a few weeks later, OpenAI actually announced that they will discontinue the support for Codex API — which includes our davinci model.

Improvements with model gpt-3.5-turbo

We then switched our attention to testing with gpt-3.5-turbo, as the API for gpt-4. was not yet available at the time.

Results were better, we were able to get some SQL which was “human readable,” once we parsed through all the additional noise from the response.

Request:

customers who have email domain example and live in new york

Response:

As you can see above, we really wanted to test with simple requests. We know that most of our enterprise customers require much more sophisticated audiences, but we need to start somewhere to understand the ChatGPT potential.

Promising ChatGPT 4

Previous tests were completed with ChatGPT 3.5. Fast forward a few weeks, with the availability of ChatGPT 4 API, we migrated our tests to that new version and different models.

We are already able to see significant progress from our early tests as you can see with the following user prompt:

Request:

people who have net_revenue greater than 10,  purchased category B products at least once and are emailable

Response:

SELECT customer_master.customer_id
FROM customer_master
JOIN orders ON customer_master.customer_id = orders.customer_id
JOIN products ON orders.product_id = products.product_id
WHERE customer_master.email_is_valid = 1
AND products.product_category = 'B'
GROUP BY customer_master.customer_id
HAVING SUM(orders.sale_amount) > 10;

After some more tweaking, we have been able to display these promising results directly in ActionIQ.

Top 3 Learnings: AI as a Copilot is Coming, but Not So Fast

The premise: Marketers use natural language to describe an audience, use ChatGPT figures out how to write a SQL query to generate an audience using the client’s data.

In practice, the progress in the performance of LLMs is undeniable and things are moving fast, but these are not yet ready for broader adoption and usage in a prod environment for our business users.

Our top three learnings from this experience:

1. Input and memory limitations. ChatGPT has input limits and no memory persistence. We do expect the input size to continue growing in the future but, until then, any real customer data would take more than the current limit to run.
2. Schema clarity. ChatGPT does fairly well guessing which columns to use in our examples, because the column names are intuitive. In the real world, we know that clients have tables t_c_all and columns column1 which would be very challenging to translate.
3. Hallucinations. At times, ChatGPT would provide back some responses looking like a SQL query at first sight, but far from being anything usable or relevant with the original request.

ChatGPT for Business Takeaway: The Need for Custom and Specialized Models

OpenAI models are trained on open web data. These models did incredibly well in democratizing access and understanding of powerful capabilities for users. And today, they are great for demos.

But the future success in enterprise organizations will depend on purpose-built models, rather than generic ones.

Databricks understood that well with their recent plan to acquire MosaicML. Allowing organization to access and customize models on their own data is what will create a true differentiation to significantly improve their business operations and reach their goals.

Despite the progress, honest people will admit that none of this is ready today for broad adoption. There are still challenges along the way such as privacy, security and costs.

What’s Next With ActionIQ and AI?

Supporting the creation of audience segments is only one of many opportunities to ease the life of both technical and business users around existing ActionIQ workflows. Imagine, for example, using ChatGPT in the opposite way of what we aimed to achieve today: reverse engineering an audience and explaining a complex definition in simple, plain english words.

If you’ve followed the evolution of the customer data platform (CDP) space, you’ve certainly heard about the benefits of deploying ActionIQ in a composable architecture. Which makes the moves around AI from Databricks and other vendors even more exciting, we can help accelerate the democratization of AI across the organization, not just of IT and data teams.

ActionIQ is committed to continuing to innovate and explore new ways to support clients with the customer data stack and customer experience use cases. AI might not be ready for you today, but it will be in the future, and when it will be, so will we, we will be sure to provide you with its power.

In the meantime, receive a custom demo to learn how ActionIQ can help your organizations maximize the value of your existing data stack and increase revenue generated via superior customer experiences.