Build an AI‑Powered Meeting Summarizer with Zoom SDK, Whisper, and OpenAI

Why AI Meeting Summaries Matter

Imagine finishing a 90‑minute strategy call and instantly receiving a concise recap, a list of decisions, and clear next steps. According to a 2023 Gartner survey, knowledge workers spend an average of 2.5 hours per week rewriting meeting notes, a productivity drain that AI can eliminate. An AI meeting summarizer not only saves time but also reduces miscommunication by capturing every spoken detail in a searchable format.

Beyond convenience, real‑time meeting analytics enable managers to track participation, sentiment, and action‑item completion rates. When these insights are powered by reliable transcription and natural‑language understanding, organizations can turn every conversation into a data point for continuous improvement.

Setting Up the Development Environment

Before writing any code, confirm that your workstation meets the following minimums: Ubuntu 20.04+ or macOS 12+, Python 3.9+, Node.js 16+, and at least 8 GB of RAM. The Whisper model consumes roughly 1.5 GB of VRAM for the base version, so a GPU with 4 GB memory is recommended for low‑latency transcription.

Install the core dependencies with a single command line. The example below uses a Bash block that creates a virtual environment, upgrades pip, and installs the required libraries.

python3 -m venv venv && source venv/bin/activate && pip install --upgrade pip && pip install openai==1.2.0 zoomus==2.0.0 git+https://github.com/openai/whisper.git

After the installation finishes, verify the versions: python -c "import whisper, openai, zoomus; print(whisper.__version__, openai.__version__)".

Integrating Zoom SDK for Live Audio Streams

The Zoom Meeting SDK provides a secure WebSocket that delivers raw audio packets in real time. Register an OAuth app in the Zoom Marketplace, enable the “Meeting SDK” scope, and copy the client ID and secret. Store these credentials in environment variables ZOOM_CLIENT_ID and ZOOM_CLIENT_SECRET.

The following Python snippet creates a meeting client, joins a meeting, and subscribes to the audio stream. Note the use of async functions to keep the pipeline non‑blocking.

import os, asyncio, json, base64, zoomus
from zoomus import ZoomClient

client = ZoomClient(os.getenv("ZOOM_CLIENT_ID"), os.getenv("ZOOM_CLIENT_SECRET"))

async def on_audio(packet):
    # packet["audio"] is a base64‑encoded PCM frame
    audio_bytes = base64.b64decode(packet["audio"]) 
    await transcribe_queue.put(audio_bytes)

async def join_meeting(meeting_id, password):
    response = client.meeting.join(meeting_id=meeting_id, password=password)
    # Assume response contains a WebSocket URL for audio
    ws_url = response["audio_ws_url"]
    async with websockets.connect(ws_url) as ws:
        async for message in ws:
            await on_audio(json.loads(message))

# Example usage
# asyncio.run(join_meeting("123456789", "myPass"))

Replace the placeholder meeting_id and password with real values during testing. The audio packets are now ready for Whisper.

Transcribing with Whisper

OpenAI’s Whisper model excels at noisy environments, making it ideal for Zoom calls where background chatter is common. Load the base model once at startup to avoid repeated disk I/O.

The transcription worker pulls raw PCM frames from the transcribe_queue, assembles them into 30‑second chunks, and calls Whisper’s transcribe method. The result is a plain‑text string that can be streamed to downstream processors.

import whisper, asyncio

model = whisper.load_model("base")
transcribe_queue = asyncio.Queue()
summary_queue = asyncio.Queue()

async def transcribe_worker():
    buffer = b""
    while True:
        chunk = await transcribe_queue.get()
        buffer += chunk
        if len(buffer) >= 30 * 16000 * 2:  # 30 seconds of 16kHz 16‑bit audio
            result = model.transcribe(buffer)
            await summary_queue.put(result["text"]) 
            buffer = b""

# asyncio.create_task(transcribe_worker())

Adjust the sample rate and byte depth if your Zoom SDK delivers a different format. The summary_queue now holds raw transcripts ready for AI‑driven analysis.

Extracting Action Items with OpenAI

OpenAI’s GPT‑4o model can parse a transcript and return a structured JSON list of action items, owners, and deadlines. Prompt engineering is crucial: a well‑crafted instruction reduces hallucinations and improves consistency.

Below is a Python function that sends the transcript to the OpenAI Chat Completion endpoint and extracts the JSON payload. The temperature is set to 0 to favor deterministic output.

import os, json, openai

openai.api_key = os.getenv("OPENAI_API_KEY")

SYSTEM_PROMPT = "You are an assistant that extracts actionable tasks from meeting transcripts. Return a JSON array with fields: task, owner, due_date (ISO 8601 or null)."

async def extract_actions(transcript):
    response = await openai.ChatCompletion.acreate(
        model="gpt-4o",
        temperature=0,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": transcript}
        ]
    )
    json_text = response.choices[0].message.content.strip()
    try:
        return json.loads(json_text)
    except json.JSONDecodeError:
        return []

Integrate this function into the pipeline by consuming items from summary_queue, calling extract_actions, and storing the result in a database or pushing it to a Slack channel.

Real‑Time Analytics and UI Considerations

For a responsive dashboard, use a WebSocket server (e.g., Socket.IO) to broadcast transcription snippets and extracted tasks to the front end. The UI can display three panels: live transcript, summarized bullet points, and a sortable table of action items.

Performance metrics matter: aim for sub‑5‑second latency from spoken word to displayed action item. Cache the Whisper model in memory, batch audio frames, and limit OpenAI calls to one per 30‑second segment to stay within rate limits.

Testing and Deployment

Write unit tests for each pipeline stage. For Whisper, mock the model’s output with a static string; for OpenAI, use the openai-mock package to simulate JSON responses. End‑to‑end tests should spin up a local Zoom meeting using the Zoom API’s “Create Meeting” endpoint, join with the SDK client, and verify that the final JSON contains expected tasks.

When deploying to production, containerize the service with Docker. A minimal Dockerfile copies the virtual environment and sets the entry point to a script that launches the async event loop. Remember to expose ports 8000 (WebSocket) and 443 (HTTPS) and to store secrets in a vault rather than hard‑coding them.

Conclusion

Building an AI‑powered meeting summarizer blends three powerful components: Zoom’s real‑time audio feed, Whisper’s high‑accuracy transcription, and OpenAI’s language understanding. By wiring them together with asynchronous Python, you can deliver live summaries, actionable task lists, and analytics that turn every call into a measurable outcome. The key takeaways are to secure the SDK credentials, keep the transcription pipeline lean, and craft precise prompts for action‑item extraction.

With the code snippets and deployment checklist provided, you are ready to prototype, iterate, and scale a solution that saves hours of manual note‑taking for any organization.

Sources

• Zoom Meeting SDK Documentation – developer.zoom.us
• OpenAI Whisper GitHub Repository – github.com/openai/whisper
• OpenAI API Reference – platform.openai.com/docs/api-reference

Author: Mahmut Sarıkaya — sarikayadev.com