Voice To Instrument idea - looking for developers.

DSP, Plugin and Host development discussion.
Post Reply New Topic
RELATED
PRODUCTS

Post

- Introduction:

The plugin we want to create is a plugin that can change any input sound in any instrument you want to choose real-time. For example if you have a voice recording, guitar recording or any other melodic instrument the producer gets the option to transform this recording into a different instrument realtime. This way people can put their ideas directly into a good sounding instrument without too much effort or having to buy/learn a new expensive instrument. We see this would be something new and revolutionary in musical creation.


- To achieve this we could go for several ways of processing.

One way would be to use the actual sound of the input to change it through manipulating the formant, pitch and adding convolution and noises to transform the actual input to the timbre of the chosen sound by costumer. For example, we've made a version in FL Studio's patcher that is able to do this a bit to change your voice in an actual flute sound. First the audio goes through a pitch and formant changer to get the tone we want. And in parallel the audio goes to a noise generator which catches the audio input and only plays when audio is coming in. That noise generator has a sample layer with the sound of a blowing noise on a panflute with a couple of velocity layers. This noise goes through a convolution reverb. (This convolution was made with an impulse response going through an actual pan-flute which gives the noise the room of the actual pan-flute. Then we recorded a panflute and did some EQ matching, and put that preset onto the parallel signal from the pitched and formanted audio and the noise. With some extra equalisation and reverb we got pretty close to the sound of an actual pan-flute. We could do this with more instruments as well, but we would be limited. A piano sound will be really hard to imitate as a piano is a snare instrument. I have made a video which demonstrates this. Here; https://www.youtube.com/watch?v=_1k3FKY ... e=youtu.be I think the 9 series of the Electro Harmonix pedals do the same. But they also do something interesting with detecting polyphonic pitch in real time as well. This could be interesting to take a look at as well. I tried reaching out to them to talk about our idea of putting that in a plugin, but we didn't get any response yet. Demo: https://youtu.be/MiTHc8d9D20?t=290 Melda Character also has a way of combining instruments with an input of a different signal.

Another option would be to replace a millisecond of the input of the waveform in real time to the waveform of a piano/flute or any instrument selected. This way you could sound like any instrument. I don't have the knowledge to explain how we could achieve this but I think it would be the best way to come close to replicate the actual sound of an instrument.

The third option would be a little more complicated, but also interesting. Changing the audio input to "midi 2" in real time.. This would be interesting if we could get it working to let the VST connect to a 3rd party instrument or our own instrument which would then be shaped into a program that's specified for changing your voice into instruments. Kind of what ROLI does with their instruments. A program called IMITONE has this idea to change a vocal to a midi instrument, but this is very static as it is MIDI 1 and does not record any human pitch changes as vibrato or just when a note is not on the nose. Also this is not a VST plugin. https://imitone.com/


- My preference would be a simple layout (which we are able to design ourselves) with the first, or second development method (changing the waveform to the waveform of an instrument to your choice) if this is possible in any way. We are able to do good promotion if this idea is statistically reachable.



We are willing to split the take or pay an amount upon agreed.



Please reach out to us if you're interested:
robbert@stars-agency.com
mellophonerecords@gmail.com

Post

Wow. Where to begin....your voice to flute is "tuned" to the voice you are using not some general "voice" -I really admire your ambition but this looks like a MASSIVE amount of work and even then looks "mostly not do-able".
VST/AU Developer for Hire

Post


Post

I did some monophonic audio to midi a few times so long ago I can't recall many details.

Am not trying to offer advice on how to do it. Am sure there are numerous ways to do it.

I lost interest in being able to sing a sax line or whatever because if the tracking was even half decent, regardless whether picked a sax or violin or guitar or synth dest patch
It didn't sound like sax. It sounded like singing with a sax tone. It didn't sound like violin, but only sounded like singing with a violin tone. With voice input, basically any patch sounded like high tech kazoo.

So the dream of a guy who can't play and can barely sing, humming in all the parts of his mental symphony, would need rather vast amounts of some kind of AI to hide the vocal stylistic signature.

If all you have is good tracking plus maybe some simple autocorrection- Even with the finest synthetic orchestral timbres, it would sound like a kazoo symphony.

The only singers who could sing a convincing sax or violin, unless assisted by some kind of sophisticated process to remove and morph the vocal stylistic characteristics, would be very rare excellently accomplished vocalists who can sing as precise as a good saxophonist, violinist, guitarist or keyboardist. A singer having uncanny superhuman precision. Ability for instance completely avoid his natural vocal vibrato while simultaneously emulating natural violin or sax vibrato. Etc.

Very few people could sing good enough to succsessfully mimic other instruments. A terrible singer would sound terrible even with the finest sax or violin patch.

Same as a good tracking midi guitar. A killer analog synth patch or sax or violin generally sounds like guitar if it tracks good enough to capture the guitar's pitch envelopes and dynamics. And if it doesn't track good enough to follow a player's exprssion then what good is that? Defeats the design goal.

Needs some strong AI or whatever mojo to strip out the source audio stylistic mannerisms and substitute equivalent destination instrument stylistic mannerisms. So that the singing sax doesn't sound like kazoo and the midi guitar sax doesn't sound like guitar with quirky fuzz box.

Post

JCJR wrote: Sat Nov 30, 2019 4:20 am I did some monophonic audio to midi a few times so long ago I can't recall many details.

Am not trying to offer advice on how to do it. Am sure there are numerous ways to do it.

I lost interest in being able to sing a sax line or whatever because if the tracking was even half decent, regardless whether picked a sax or violin or guitar or synth dest patch
It didn't sound like sax. It sounded like singing with a sax tone. It didn't sound like violin, but only sounded like singing with a violin tone. With voice input, basically any patch sounded like high tech kazoo.

So the dream of a guy who can't play and can barely sing, humming in all the parts of his mental symphony, would need rather vast amounts of some kind of AI to hide the vocal stylistic signature.

If all you have is good tracking plus maybe some simple autocorrection- Even with the finest synthetic orchestral timbres, it would sound like a kazoo symphony.

The only singers who could sing a convincing sax or violin, unless assisted by some kind of sophisticated process to remove and morph the vocal stylistic characteristics, would be very rare excellently accomplished vocalists who can sing as precise as a good saxophonist, violinist, guitarist or keyboardist. A singer having uncanny superhuman precision. Ability for instance completely avoid his natural vocal vibrato while simultaneously emulating natural violin or sax vibrato. Etc.

Very few people could sing good enough to succsessfully mimic other instruments. A terrible singer would sound terrible even with the finest sax or violin patch.

Same as a good tracking midi guitar. A killer analog synth patch or sax or violin generally sounds like guitar if it tracks good enough to capture the guitar's pitch envelopes and dynamics. And if it doesn't track good enough to follow a player's exprssion then what good is that? Defeats the design goal.

Needs some strong AI or whatever mojo to strip out the source audio stylistic mannerisms and substitute equivalent destination instrument stylistic mannerisms. So that the singing sax doesn't sound like kazoo and the midi guitar sax doesn't sound like guitar with quirky fuzz box.

AI is definitely an option too, though we have no idea in detail how we could achieve this. Though we have ideas on how to make this possible with midi 2.

Post

Lind0n wrote: Fri Nov 29, 2019 5:44 pm Wow. Where to begin....your voice to flute is "tuned" to the voice you are using not some general "voice" -I really admire your ambition but this looks like a MASSIVE amount of work and even then looks "mostly not do-able".
Yes, you're absolutely right. This will take loads and loads of work. But even though it's really ambitious I think it would be groundbreaking if it's executed well. I could develop a new compressor but that wouldn't move anything forward.

:) Think big! Sometimes ideas are far from reachable, but sometimes it works out 8)
If I would ask you 10 years ago about that I wanted to create a plugin that could put the vocals down on a master buss it would be nonsense.. And now that's reality too! (izotope ozone 9)

Post

So something similar to Abletons Audio To Midi but in real time? Best of luck with the development :)

Post

vandalenmusic wrote: Sat Nov 30, 2019 6:50 pm AI is definitely an option too, though we have no idea in detail how we could achieve this. Though we have ideas on how to make this possible with midi 2.
I don't have a good idea how to do the "stylistic mapping" either. I just think such a feature would be needed before a voice tracker would be generally musically useful except for rare extremely talented vocalists capable of accurately singing "in the style of" a target instrument.

It has been a long time since I did mono pitch tracking but realtime pitch tracking is not a real tough problem. I mean I'm a dullard and got it working not so bad decades ago so a smart person could do it lots better. Roland was doing realtime midi guitar since the 1980's and guitar is tougher than vocal humming. Arp avatar even earlier, though that was kinda rough, but done with just analog and cmos digital gates, not even a cpu. Avatar would probably have worked better if it didn't have to also decide which of 6 strings to monophonic track without even a cpu.

There were earlier entirely analpg homebrew pitch to CV using phase lock loops. But even converting vocal humming to CV, then driving VCO and VCF and VCA mono analog synth, maybe even transpose the pitch higher or lower than the vocal range.

Strangely if it tracks anywhere near decent, it doesn't adequately anonymize the vocal source stylistic gestures. Tends to sound like analog synth kazoo! Just ultra cheezy to my ear.

Post

JCJR wrote: Sat Nov 30, 2019 4:20 am It didn't sound like sax. It sounded like singing with a sax tone. It didn't sound like violin, but only sounded like singing with a violin tone. With voice input, basically any patch sounded like high tech kazoo.

So the dream of a guy who can't play and can barely sing, humming in all the parts of his mental symphony, would need rather vast amounts of some kind of AI to hide the vocal stylistic signature.
^ Very relevant

At one point I wanted to make a "synth with voice input", and was driving an oscillator with the detected pitch. Even if you remove the formant variations, the pitch variations strongly remind of the voice, also the envelope of course. So I'm behind the "high-tech kazoo" moniker :lol:
Checkout our VST3/VST2/AU/AAX/LV2:
Inner Pitch | Lens | Couture | Panagement | Graillon

Post

vandalenmusic wrote: Sat Nov 30, 2019 6:55 pm
Lind0n wrote: Fri Nov 29, 2019 5:44 pm Wow. Where to begin....your voice to flute is "tuned" to the voice you are using not some general "voice" -I really admire your ambition but this looks like a MASSIVE amount of work and even then looks "mostly not do-able".
Yes, you're absolutely right. This will take loads and loads of work. But even though it's really ambitious I think it would be groundbreaking if it's executed well. I could develop a new compressor but that wouldn't move anything forward.

:) Think big! Sometimes ideas are far from reachable, but sometimes it works out 8)
If I would ask you 10 years ago about that I wanted to create a plugin that could put the vocals down on a master buss it would be nonsense.. And now that's reality too! (izotope ozone 9)
I think a couple of people here have replied with the issue MOST at hand. The stylistic and sonic limitations imposed by any given instrument are in fact part of its "fingerprint" and attempting to make a realistic violin passage using a tuba fails because of this - never mind the sonic issues, and the real-time midi tracking, pitch detection etc. etc.

I wish you the best of luck.
VST/AU Developer for Hire

Post Reply

Return to “DSP and Plugin Development”