Would be possible an version like MoE below 5B?

#9
by JoaoZaokk - opened

Im training this model, and its AMAZING how well it works with new capabilities, computer use gone from almost none to 80 percent with just some million tokens, desktop was about 30% coming from zero, but web navigation had a BIG jump.
Screenshot to page, ~30% to 100%, i doubt this number, but the results are incredible, it naturally started using tailwind, and completed the whole pages in 80% of the training.

Within the same gpu, blocked sliding windows attention + per-layer compile got 19.7k. Before it was 6k tokens. All that on a rented rtx pro 6000.

Ive maded a test to transform it on an MoE, but, after a few days and almost a billion tokens i couldnt moetize it. But it would be amazing to think, how fast could a model light like that, with the speed of the experts?

Sign up or log in to comment