In this post
First Self-Serve Platform for Fine-tuning ModernBERT
Late December 2024, Answer.ai and LightOn announced the release of ModernBERT, a significant improvement across older encoders, with faster processing and support for 8192 sequence lengths.
Today, we're thrilled to announce support for ModernBERT fine-tunes. You can now use the AutoEval platform to fine-tune your own ModernBERT model with only a few clicks.
Why ModernBERT?
Encoder style models like DeBERTa have had the practical purpose of having contextual understanding to perform well on a variety of NLP tasks, such as Text Classification, Entity Recognition, and Question Answering. With the interest in Generative AI, the ML world has seen a number of innovations with better & cleaner training data, better embeddings, and underlying library improvements.
The folks from Answer.ai, LightOn, and friends from HuggingFace and MosaicML have brought ModernBERT to the modern age of ML. They've improved the training data, employed rotary positional embeddings (RoPE), expanded the context length, and many more advancements. And now, the LastMile AI team is proud to announce having it readily available for anyone to fine-tune on our platform.
How do I get started?
Login to your LastMile AI account
Create/Upload a dataset (example: Downsampled SNLI, SNLI)
Fine-tune a metric from the dataset
Select ModernBERT-Large in the model list
Voila! You have your own fine-tuned ModernBERT on your own data.
What's next?
We're doing post-training on the ModernBERT model to improve accuracy for LLM evaluations and improve performance on longer context lengths. Stay tuned for more updates and reach out to us at team@lastmileai.dev if you want a sneak peak!
