Skip to content

Break audio up into chunks before sending to google #51

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
albertreed wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Break audio up into chunks before sending to google #51

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions components/conversation.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@ const SCREEN_PLAYING = ScreenOutConfig.ScreenMode.PLAYING;
const SCREEN_OFF = ScreenOutConfig.ScreenMode.OFF;
const DEFAULT_SAMPLE_RATE_IN = 16000;
const DEFAULT_SAMPLE_RATE_OUT = 24000;
// The max number of bytes of audio that can be sent in a single chunk.
// Not documented anywhere by google, but sending more than this amount
// throws an INVALID_REQUEST error stating that audio_in is too long.
const MAX_FRAME_LENGTH = 31 * 1024;
Copy link
Copy Markdown
Owner

@endoplasmic endoplasmic May 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this just figured out by process of elimination? Love me the prime number :D

Copy link
Copy Markdown
Author

@albertreed albertreed May 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sadly, yes. I'm very unfamiliar with the gRPC stuff so if you know a better way, let me know. It's susceptible to changes on their end which is obviously bad.


let conversationState;
let volumePercent = 100;
Expand Down Expand Up @@ -203,8 +207,16 @@ function Conversation(assistant, config) {
// if audio tries to come in when we are sending text, bail out
if (sendingText) return;

const request = AssistRequest.create({ audioIn: data });
conversation.write(request);
// Take the audio data and split it up into bite-sized chunks that google can digest
// and then send it along. Under ordinary circumstances, we should be capturing
// audio data fast enough to send in a single request so that the assistant can be
// highly responsive.
let chunks = [];
while(chunks.length < data.length / MAX_FRAME_LENGTH) {
chunks.push(data.slice(chunks.length * MAX_FRAME_LENGTH, Math.min(data.length, (chunks.length + 1) * MAX_FRAME_LENGTH)));
Copy link
Copy Markdown
Owner

@endoplasmic endoplasmic May 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I've never done reviewing in github, so forgive my noobness)

Can you break your Math.min into a const and give it a name explaining what it's doing?

}
chunks.forEach((chunk) =>
Copy link
Copy Markdown
Owner

@endoplasmic endoplasmic May 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you use the curly braces to wrap the for each, or put it all on 1 line?

conversation.write(AssistRequest.create({ audioIn: chunk })));
};

// end the conversation
Expand Down