top of page

LBSocial

OpenAI Responses API: Advanced Tweet Analysis with File & Web Search Integration

Writer: Xuebin WeiXuebin Wei
Mar 203 min read

Updated: 1 day ago

 



This blog demonstrates how to use the OpenAI Responses API to analyze Twitter data by leveraging file search, web search, and stateful responses to efficiently retrieve both private and real-time public information.



Last week, OpenAI released the Responses API, a powerful upgrade that merges the functionalities of the Chat Completions API and the Assistants API into a unified system. This API enables developers to:

  • Perform traditional chat completions like in GPT models.

  • Use web search to retrieve real-time information from the internet.

  • Conduct file search on private datasets using vector stores.


Since the Assistants API will be retired in 2026, OpenAI now recommends using the Responses API instead.


Setting Up the Environment

To begin, we install the required libraries:

pip install openai pymongo -q

In this tutorial, we retrieve tweets from a MongoDB database, process them into a structured format, and use file search and web search with OpenAI’s Responses API.


Connecting to MongoDB & Extracting Tweets

For this demonstration, we use MongoDB to store and retrieve Twitter data. First, we connect to the database and extract tweets that contain discussions on Generative AI.


Once extracted, the tweets are converted into JSON format and stored in memory, allowing seamless embedding and retrieval of data.


Initializing OpenAI Responses API

With our tweet dataset ready, we initialize the OpenAI Responses API to interact with the vector store and retrieval tools.


To do this, we authenticate using an API key stored securely in AWS Secrets Manager, ensuring our credentials remain protected.


Using File Search for Private Twitter Data

The File Search API allows us to analyze stored tweets efficiently by creating a vector store, uploading data, and retrieving relevant insights.


Run File Search

file_search_response = client.responses.create(
    model="gpt-4o",
    input="latest developments in Generative AI",
    tools=[{"type": "file_search", "vector_store_ids": ["your_vector_store_id"]}]
)
print(file_search_response.output_text)

This retrieves relevant tweets from our private dataset stored in OpenAI’s vector store.


Using OpenAI Web Search API for Real-Time Data

While file search helps analyze private datasets, sometimes we need real-time insights.

This is where OpenAI’s Web Search API becomes powerful—it enables the model to fetch the latest updates from the internet.


Run Web Search

web_search_response = client.responses.create(
    model="gpt-4o",
    input="latest developments in Generative AI",
    tools=[{"type": "web_search"}]
)
print(web_search_response.output_text)

Maintaining Conversation Context with Stateful Responses

A unique feature of the Responses API is stateful responses, allowing AI to maintain conversation history across multiple interactions.


For example:

  1. Initial Query: “Tell me about Generative AI.”

  2. Follow-up Query: “Find different news sources.”

By linking the second query to the first using a response ID, the model remembers previous searches, ensuring more context-aware results.


Combining File Search & Web Search for Enhanced AI Retrieval

To get the best of both worlds, we can merge file search and web search in a single query, ensuring comprehensive results from both private and public data.


Run Combined Search

combined_search_response = client.responses.create(
    model="gpt-4o",
    input="latest advancements in Generative AI",
    tools=[
        {"type": "file_search", "vector_store_ids": ["your_vector_store_id"]},
        {"type": "web_search"}
    ]
)
print(combined_search_response.output_text)

This method allows us to compare Twitter discussions with real-world updates, enhancing retrieval-augmented generation (RAG) applications.


Conclusion

In this tutorial, we explored how to use the OpenAI Responses API for AI-driven Twitter data analysis, including:

  • File Search for retrieving insights from stored tweets.

  • Web Search for real-time public information.

  • Stateful responses for maintaining conversation history.

  • Combining File & Web Search for a more powerful AI-driven retrieval system.


With the Responses API, developers can build smarter, context-aware, and AI-powered data retrieval applications.



 
 
 

Comentarios

bottom of page