experiment

Header

Footer

PreFooter

SectionTitle

PaginationButton

BlogPage

Trusted by world class companies, Scale delivers high quality training data for AI applications such as self-driving cars, mapping, AR/VR, robotics, and more.

AI & ML Blog | Scale AI

NextImage

AnnouncementContextProvider

Screen

plasmic-graphcms

scale.com (Production)

lottie-react

plasmic-nav

react-scroll-parallax-global

plasmic-embed-css

plasmic-query

plasmic-basic-components

plasmic-tabs

react-aria

plasmic-cms

react-chartjs-2

plasmic-strapi

react-slick

radix-ui

react-quill

Product

Company

General

People

Engineering

Scale AI, the data infrastructure for AI, has been named a Cool Vendor in the 2022 <a href="https://www.gartner.com/doc/reprints?id=1-2C7YMC4Z&amp;ct=230109&amp;st=sb">Gartner Cool Vendors&trade; in Data-Centric AI</a> by Svetlana Sicular, Arun Chandrasekaran, Sumit Agarwal, Oleksandr Matvitskyy and Anthony Mullen.
The biggest barriers for organizations looking to implement data-centric AI strategies are having the necessary resources, talent, experience and high-quality data needed to feed algorithms and leverage the power of AI. Scale provides AI solutions tailored to business use cases and machine learning tools to unlock breakthroughs like Generative AI and operationalize AI for all organizations spanning the U.S. government, researchers, startups and Fortune 500 companies across e-commerce, logistics, technology, fintech, and more industries. Organizations such as Microsoft, Open AI, SAP, Flexport and the U.S. Army partner with Scale to solve problems with data labeling and annotation, scenario-based model testing and validation, ​​content understanding and contextualization, AI catalog for asset reusability and more.
According to Gartner, &ldquo;Data challenges plague AI implementations. They are often unexpected because AI teams focus on algorithms, and data concerns come as an afterthought.&rdquo; It recommends that when evaluating data-centric AI solutions, organizations should &ldquo;work with stakeholders to determine what data characteristics are high value for AI,&rdquo; and &ldquo;add AI capabilities such as AI-specific data management, synthetic data and data labeling &ndash; to your data management toolbox.&rdquo;
&ldquo;It&rsquo;s an honor to be named a Data-Centric AI Cool Vendor by Gartner. We believe that it is a testament to Scale&rsquo;s mission to accelerate the development of AI applications and showcases the market impact of our vision,&rdquo; said Alexandr Wang, CEO and Founder of Scale AI. &ldquo;The solutions that we provide enable organizations to reap the benefits of AI, whether they are building or applying AI. This means better business outcomes, a real return on investment, and a faster time to production.&rdquo;
According to the Gartner report, &ldquo;By 2025, 90% of new data and analytics deployments will be through an established data ecosystem, causing consolidation across the data and analytics market.&rdquo; Therefore, introducing a data management approach, such as data quality, data diversity, data and analytics governance, and data literacy to AI teams now, is imperative to any organization's AI strategy.
Download the full report <a href="https://www.gartner.com/doc/reprints?id=1-2C7YMC4Z&amp;ct=230109&amp;st=sb">HERE</a>.
Gartner Disclaimer
GARTNER and COOL VENDORS are the registered trademarks of Gartner Inc., and/or its affiliates and has been used herein with permission. All rights reserved. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner&rsquo;s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.
About Scale AI
Scale accelerates AI development by providing the data infrastructure for AI. Its data-centric platform manages the entire ML lifecycle, from data annotation and curation to model testing and evaluation, enabling any organization &ndash; from the world&rsquo;s most advanced AI teams to legacy organizations &ndash; to develop and deploy impactful AI solutions. Scale combines ML technology with skilled human insight to ensure every AI application is built on a foundation of high-quality ground truth data. Scale is trusted by the largest technology companies in the world, including Microsoft, Toyota, Nuro, Samsung, Square, PayPal, SAP, Instacart, Pinterest, Brex, Flexport and U.S. government agencies like the U.S. Army and Air Force. Founded in 2016 and valued at $7.3 billion, Scale is backed by Founders Fund, Accel, Index, and Tiger Global.

image.jpg

Scale AI, the data infrastructure for AI, has been named a Cool Vendor in the 2022 Gartner Cool Vendors™ in Data-Centric AI.

Scale AI Named a 2022 Cool Vendor by Gartner® | Scale AI

Scale AI Named a 2022 Cool Vendor by Gartner®

Scale AI is dedicated to helping the Department of Defense (DoD) compete, fight, and win by providing high-quality, government-owned data that can power any AI/ML and autonomy effort.&nbsp;A first step towards this goal is organizing the DoD data layer. Technologies still relatively new for the DoD - like cloud computing and data repositories - are a good start. These are insufficient however, to provide the DoD with data processing capabilities required to fuel large-scale autonomy operations.&nbsp;Scale can provide the next critical layer in the software stack which can bring autonomy efforts out of the labs, into production, and onto the battlefield.
Beyond developing, training, and deploying individual autonomous systems, taking the time to &ldquo;get the data layer right&rdquo; can have broader impact, such as enabling the creation of &ldquo;Data Annexes&rdquo; to go along with Operational Plans (OPLANs).&nbsp;These would allow DoD leaders to provide the US Army&rsquo;s autonomous vehicle fleets with key, pre-validated datasets for critical operations in theater. These would include theater-specific perception models (LiDAR, camera, radar), segmentation-mapped avenues of approach, Order-of-Battle class and type IDs for detection and recognition, EO and SAR imagery mapped to friendly schemes of maneuver, and much more. With a single update, a vehicle fleet would be prepared to move out.&nbsp;
Scale provides the data and machine learning operations (MLOps) pipelines for the world&rsquo;s most ambitious robotics and autonomy teams.&nbsp;As the DoD explores novel approaches to human-machine teaming, it will require a sophisticated, government-owned data strategy with dedicated tooling.&nbsp;This will provide the numerous vehicle platforms and programs with the highest quality data to drive production-grade autonomy. Getting this right is a precursor to mission success.&nbsp;
Scale provides an open data architecture and MLOps pipeline in alignment with the DoD Modular Open Systems Approach (MOSA), decoupled from vehicle ownership.&nbsp;This allows the US Government to reuse training data and algorithms developed within Scale platforms for other programs. Thus the DoD can create a &ldquo;data flywheel&rdquo; that strengthens as more programs develop production-grade autonomy.
<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fceb82bb90c4741f2a98cdc32b2de19a4">
Scale currently executes billions of <a href="https://scale.com/guides/data-labeling-annotation-guide" target="_blank" rel="noopener noreferrer">annotations</a> per month, maps 3,800 kilometers of roads per week, and leads the industry as the AI perception data provider. Scale partners with self-driving vehicle companies training and building systems ranging from small delivery vehicles to long-haul trucks. Our partners include Kodiak Robotics, Voyage, Optimus Ride, Toyota Research Institute, Embark, General Motors, Honda innovations, Oshkosh, and Nuro - the first autonomous ground delivery vehicle to receive <a href="https://scale.com/customers/nuro" target="_blank" rel="noopener noreferrer">DoT approval</a> for public roads. Additionally, Scale is helping to develop autonomy in other unique domains, including maritime vessels (<a href="https://scale.com/customers/sea-machines-robotics" target="_blank" rel="noopener noreferrer">Sea Machines</a>), autonomous drones (<a href="https://scale.com/customers/skydio" target="_blank" rel="noopener noreferrer">Skydio</a>), and autonomous warehouse systems (<a href="https://scale.com/customers/otto-motors" target="_blank" rel="noopener noreferrer">OTTO Motors</a>).&nbsp;
Through our in-house capabilities, direct partnerships, and an extensive subcontractor network; Scale collects, curates, and <a href="https://scale.com/guides/data-labeling-annotation-guide" target="_blank" rel="noopener noreferrer">processes</a> more ground truth perception data than any other company in the world.
We are a $7B company with real-world applications at high levels of mission-readiness. Scale has over 700 employees focused on cutting edge ML development, human-machine teaming, content understanding, and novel applications for machine learning in the commercial and federal sectors. Our Modular Open System Architecture reduces cost and time to deployment, increases opportunities for iteration, and allows the government to pick and choose the tools and data it needs, all while maintaining full data rights.
Jonathan Hudgins is a member of the Federal Team at Scale where he focuses on bringing Scale&rsquo;s proven commercial solutions to solve Federal Government problems. Jonathan is a former active duty Air Force officer and F-15E Fighter Pilot.

Autonomy Table Stakes:  DoD’s Data Layer

On November 28th, OpenAI released a new addition to the GPT-3 model family: <code>davinci-003</code>. This latest model builds on <a href="https://openai.com/blog/instruction-following/" target="_blank" rel="noopener noreferrer">InstructGPT</a>, using reinforcement learning with human feedback to better align language models with human instructions. Unlike <code>davinci-002</code>, which uses supervised fine-tuning on human-written demonstrations and highly scored model samples to improve generation quality, <code>davinci-003</code> is a true reinforcement learning with human feedback (RLHF) model. It is trained with <a href="https://openai.com/blog/openai-baselines-ppo/" target="_blank" rel="noopener noreferrer">PPO</a> to optimize the generated text&rsquo;s score against a separate &ldquo;reward model&rdquo;, which is trained on rating comparisons by human graders of different model outputs. More details can be found in OpenAI&rsquo;s <a href="https://beta.openai.com/docs/model-index-for-researchers" target="_blank" rel="noopener noreferrer">model index for researchers</a>. The net result is that <code>davinci-003</code> is tuned to produce outputs that it thinks would be scored highly by humans. Scale is proud to partner with OpenAI to provide this human feedback.
OpenAI&rsquo;s announcement email mentions the following improvements for <code>davinci-003</code>:
<ul>
<li>&ldquo;It produces higher quality writing. This will help your applications deliver clearer, more engaging, and more compelling content.</li>
<li>It can handle more complex instructions, meaning you can get even more creative with how you make use of its capabilities now.</li>
<li>It&rsquo;s better at longer form content generation, allowing you to take on tasks that would have previously been too difficult to achieve.&rdquo;</li>
</ul>
But how much better is <code>davinci-003</code>, really? We decided to put it to the test quantitatively. Using <a href="https://scale.com/spellbook" target="_blank" rel="noopener noreferrer">Scale Spellbook</a>, the platform to build, compare and deploy large language model apps, we evaluated <code>davinci-003</code> versus <code>davinci-002</code> across tasks ranging from few- and zero-shot classification, summarization, and poetry writing. Here&rsquo;s what we found.
<h3>Classification</h3>
It&rsquo;s well known that <code>davinci-002</code> performs very well on classification with a few-shot prompt, and it appears that <code>davinci-003</code> offers comparable (but slightly worse) performance few-shot. Using Spellbook&rsquo;s built-in evaluation feature on 250 Yelp reviews, we compare the two models&rsquo; classification of these reviews as Positive, Negative, or Neutral in a four-shot prompt&mdash;meaning four labeled examples are included in the prompt itself.
<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F65f8dfd9c9744c15967d03341c5cdc73">
The davinci-003 model has 87.2% accuracy on this 250 Yelp review dataset sample, slightly lower than davinci-002&rsquo;s 91.6%.
However, if we ask the two models to classify these same reviews in a zero-shot prompt, we find that <code>davinci-003</code> scores more than 30% higher in accuracy than <code>davinci-002</code>&mdash;and on par with a <code>davinci-002</code> few-shot prompt! This is a substantial improvement in zero-shot classification performance. In this classification setting, adding examples to the prompt actually lowers <code>davinci-003</code>&rsquo;s accuracy.
<img src="https://lh6.googleusercontent.com/Wr30hJNdDAekBQTxtBRBN40_GR1CIElZlygaJSrAbGd6OK20V-pGS_bI5BoCF2SS4az3z7EFPYde3k5TnoVKHZd-68dwlhtHctxB7XtW5zqOXOjQkY2P8SxCkAjzlmg1PpKQ1hzEgob2W18laWD8FfSjNYShqjurevSldtMENyh8DUALgcMeYi4HQKi-">
Beyond accuracy, we found that asking <code>davinci-003</code> to explain its rationale in a zero-shot prompt works reliably, and provides a better explanation than a one-shot <code>davinci-002</code> about 75% of the time, based on Scale&rsquo;s human evaluation workforce judging a sample of 100 outputs.
<h3>Text Generation</h3>
The most notable performance leap comes in how <code>davinci-003</code> understands word and sentence structure, especially rhyming and syllables. In prompting for a poem that rhymes, <code>davinci-003</code> constructs a desired output, whereas <code>davinci-002</code> largely fails to implement rhyming.
<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F31cb06da7e1a488a91f62ef14feee823">
<code>davinci-003</code> is also better equipped to handle requests for complex output structures, such as iambic pentameter (think Shakespeare). <code>davinci-002</code> certainly provides a poem, but ignores the request for lines of 5 pairs of stressed and unstressed syllables (buh-BUM). While not perfect, <code>davinci-003</code>&rsquo;s output is significantly more faithful to the requested prompt.
<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fc3fc8edf1aea432bb7bb5118a6dc8c82">
<h3>Summarization</h3>
One practical difference between the two variants is verbosity. In our experimentation, <code>davinci-003</code> consistently generates longer outputs than <code>davinci-002</code> on the same prompts. This is most visible in summarization tasks, where the value of the result may depend on its brevity.
In a sample of 60 articles, we tested both variants&rsquo; summarization capabilities by asking them to summarize the introductory paragraphs of lengthy Wikipedia pages. The additional text generated by <code>davinci-003</code> is typically beneficial, including information central to an article&rsquo;s importance. However, the resulting summaries are also wordy. In our analysis, <code>davinci-003</code> generates outputs about 65% longer than <code>davinci-002</code> under identical prompts.
<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fbbd0ed222dff4216ab9173d07d176890">
It will be important for GPT-3 users to bear in mind <code>davinci-003</code>&rsquo;s talkativeness when choosing between the two variants. In our testing, given the task of generating a 1-sentence summary, <code>davinci-003</code> produced 42 words on average, nearly doubling <code>davinci-002</code>&lsquo;s average of 23. The shortest summary <code>davinci-003</code> gave back was a beefy 32 words, compared to only 12 words for <code>davinci-002</code>. This tendency extends to multi-sentence summaries as well.
For use cases that value brevity, <code>davinci-002</code> is the safer choice. Nevertheless, for many use cases, the longer and more detailed completions provided by <code>davinci-003</code> can be beneficial.&nbsp;
<h3>Conclusion</h3>
Overall, <code>davinci-003</code> is a meaningful improvement from <code>davinci-002</code>, especially for zero-shot classification and complex text generation.
While many of the benefits of the latest iteration of GPT-3 are apparent at the surface, concrete evaluation metrics ensure models and prompt strategies are performing as well as possible.
Luckily, Spellbook makes that easy, allowing users to run instant programmatic evaluation in classification use cases, or sending model outputs to Scale&rsquo;s high-quality workforce to get quantitative evaluation metrics for generative use cases.
<a href="https://scale.com/spellbook" target="_blank" rel="noopener noreferrer">Sign up for Spellbook today!</a>
P.S. If you&rsquo;re looking for a little dramatic flair in your emails to coworkers, <code>davinci-003</code> has you covered. It performs significantly better at creatively translating text or even explaining code as different celebrities in a zero-shot setting.
&nbsp;
BranchCommitmaintestmaintest <img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fa4f4e03e1cef4d28a44fcf465150aeca">
<pre class="ql-syntax" spellcheck="false"> Write a short rhyming poem about winter in New York davinci-002
davinci-003
</pre>
In New York City When winter descends
The air is so crisp and chilly
And the wind never ends
The snowflakes are so big and fluffy
You just want to grab a handful
And then make a snowball
People hurry to and fro
Bundled up in coats
Hats, and scarves, and gloves
No one wants to be caught
Lights are twinkling in the night, It's wintertime in New York City so bright.
The snow is glistening, the wind is cold,
The atmosphere is really bold.
The snowflakes fall with a gentle grace,
Adorning the city with a lovely lace.
The air is crisp, and the people merry,
It's a magical winter in New York City.
Text Generation
The most notable performance leap comes in how davinci-003 understands word and sentence structure, especially rhyming and syllables. In prompting for a poem that rhymes, davinci-003 constructs a desired output, whereas davinci-002 largely fails to implement rhyming.

GPT3.png

We evaluate davinci-003 across a range of classification, summarization, and generation tasks.

Evaluating davinci-003: Performance, Improvements, and Gaps

How Much Better is OpenAI’s Newest GPT-3 Model?

<h3>Background</h3> The rise of social media platforms has driven the popularity of user-generated content (UGC) and it is changing from the bottom up at a pace faster than ever. Platforms are competing for user attention and engagement - some are winning, but some are falling behind. How can a social media platform stay ahead of the game? One of the most effective ways to incentivize users to create and engage with content on the platform is by curating and promoting trends. A trend is a type of content repeated and shared amongst creators -&nbsp;a specific subject matter, action, or format that rises in popularity and eventually becomes viral. Common types of trends include dance challenges, transitions, and comedy skits. While videos in a trend follow a similar format, creators usually express their creativity and individuality by adding a personal touch to the videos. For that reason, the variety of the videos within the same trend can be wide and unpredictable. The key elements of these video trends include the video’s image, motion, audio, and text overlay. For example, a transition trend is a trend where the participants show a sudden change in their appearance, often signified by a dramatic sound effect in the background music and sometimes accompanied by dance moves, lip syncing, and text overlay. The lifecycle of a trend can vary from days to months depending on various factors. To remain competitive, social media platforms need to detect the trends while they are still emerging and put them to use immediately by matching them to the right creators and promoting them to the suitable audience.&nbsp; <a href="https://scale.com/content-understanding" rel="noopener noreferrer" target="_blank">Scale Content Understanding</a> improves platform experiences by enriching content metadata, discovering trend insights, and flagging sensitive content. One feature of Content Understanding is content data enrichment for better trend detection. To facilitate this process, we developed proprietary algorithms to detect trends early, quickly, and thoroughly.&nbsp; <h3>ML Clustering Overview</h3> <img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/449d8647-5509-470c-a6cf-ea38c14a62f3"> We took a human-in-the-loop (HitL) machine learning approach to detect trends. We represent trends as clusters of videos estimated from video-extracted embeddings. Once formed, these clusters are sent to a human pipeline (which we call “Cluster Labeling”) where taskers answer a series of questions about the videos in a given cluster - namely, if a sample of the videos corresponds to a trend and the potential name of the trend. The following are the phases of the ML Clustering process: <h3>Phase 1: Embedding Extraction</h3> To extract embeddings, we use a variety of multi-modal information extraction machine learning models, each of which is trained on large-scale, common datasets with high distribution overlap to the social media domain. <h3>Phase 2: Clustering</h3> Once the embeddings are extracted, we perform a clustering pipeline to take in these embeddings and output clusters. Overview Given the embeddings, we first perform <a href="https://arxiv.org/pdf/1802.03426.pdf" rel="noopener noreferrer" target="_blank">Uniform Manifold Approximation and Projection (UMAP)</a> dimensionality reduction (inspired by <a href="https://arxiv.org/abs/2203.05794" rel="noopener noreferrer" target="_blank">BERTopic</a>). UMAP takes in high-dimensional data and reduces it down to a lower dimensional space scalably (e.g. UMAP performs computation on tens of thousands of points on the order of tens of seconds). It reduces dimensionality in a way that preserves global Riemannian geometric structure and pairwise distances. On these low-dimensional embeddings, we perform an iterative hierarchical clustering process. Each individual point in embedding space is initially represented as a cluster.&nbsp;We then calculate the proximity between all pairs of clusters. We calculate the proximity of two clusters in a given pair as the variance in embedding space (i.e. the average sum of squares from the joined cluster mean embedding vector) of the points corresponding to if these clusters were to be joined. The two clusters that are closest to each other are joined given the above definition of proximity. This cluster-joining process is performed iteratively until all clusters are above a specified variance that is provided as a hyperparameter. <img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/24053e18-8fbc-46a8-91ba-4ea274599636"> Stop Condition Variance Hyperparameter We carefully tune the variance hyperparameter in the stop condition to balance between making the clusters too coarse or too granular. For large variance hyperparameters, the final clusters are allowed to span a wide region in embedding space, meaning they are not semantically similar enough to represent trends (which tend to be extremely specific). Diametrically opposed, for small variance hyperparameters, the clusters end up with too few videos for users in the cluster labeling pipeline to recognize the trend. We therefore carefully tune the stop condition variance hyperparameter. This provides a balance between the clusters being fine enough that they represent groups of videos that are semantically similar enough to correspond to trends, and coarse enough that users from the cluster labeling pipeline can recognize the trend. This hyperparameter is normalized in order to reduce the amount of per-dataset manual tuning. <h3>Phase 3: Cluster Post-Processing</h3> To improve the quality and throughput of the taskers, we prune out clusters that are composed of a number of videos that is smaller than our definition of a trend. We also add a post-processing density score to help taskers focus on the clusters which are tightest in embedding space - i.e. the ones that are most likely to be semantically similar. This density score is calculated as the intra-cluster variance of each cluster in the dimensionality-reduced embedding space. <h3>Conclusion</h3> Our trend detection product consistently performs at a higher than 90% recall and 90% precision in production. Our HitL ML approach enables us to stay flexible with customization towards specific content niches while maintaining a high throughput. We are able to process tens of millions of videos with a few hours of turnaround. Our priority is to not only detect trends promptly, but also equip our customers with the data and insights on these trends to take actions immediately.&nbsp; At Scale, we believe investing in data is the key to building better platform experiences and discovering trend insights. If you’re interested in learning more about <a href="https://scale.com/content-understanding" rel="noopener noreferrer" target="_blank">Scale Content Understanding</a>, speak to our team today.&nbsp;

Discovering short-form video trends is challenging. In this blog post, we explain how we use an ML + HiTL approach for faster and higher-quality trend detection.

Discovering Emerging Short Form Video Trends with Clustering and HitL

Today, we&rsquo;re excited to announce the Early Access of Scale Forge (Beta) for AI-generated product imagery in seconds. With just a few clicks, marketers and brands can create stunning visual content for ad creatives, campaigns, and social media. Learn more and <a href="https://scale.com/catalog" target="_blank" rel="noopener noreferrer">sign-up for Early Access</a>.
<h2>Unlocking Stunning Product Imagery</h2>
Currently, retailers and advertisers are limited by the amount and quality of product photography they have to create an engaging shopping experience for customers. The investment needed for photoshoots, scale of product catalogs, and the diverse tastes of audiences adds to this burden. This limits the growth of businesses, and ultimately their ability to communicate their story and unique product offering to their audience.
Building on top of our existing strengths and technology in eCommerce, we built Forge to solve this problem.
<h2>Catalog Forge</h2>
With Forge, we&rsquo;re enabling customers to create AI-generated imagery via an easy-to-use design studio. Forge uses novel ML methods to allow for high-fidelity product and brand preservation for retail products.&nbsp;
With Forge, teams can:
<ol>
<li>Experiment and prototype quickly to decide the aesthetic of images</li>
<li>Operationalize and scale product image generation to enable servicing thousands of brands and products</li>
<li>Improve conversion with enhanced product images to maximize ROI&nbsp;&nbsp;</li>
</ol>
<a href="https://scaleai.wistia.com/medias/sk212njbxf" target="_blank" rel="noopener noreferrer">Video</a>
To get started, simply provide your product image, select a scene or describe your own, and create imagery through two ways:&nbsp;
<ol>
<li>Scene Enrichment: Create unique customizable scenes to generate product images that align with your brand values and product identity. Retain the original pose of each image. Best for: rigid objects</li>
<li>Full Transformation: Experiment with different poses, zoom levels, whole scenes, complementary products, and more. Only available for select demo products right now. Best for: clothing, fashion accessories</li>
</ol>
Forge enables you to create unique and memorable product images for your brand powered by AI.
<h2>New Era of Creativity in eCommerce and Retail&nbsp;</h2>
We&rsquo;re thrilled to introduce Scale Forge for eCommerce. We&rsquo;re excited to see what companies can create once they have access to the best quality AI tools. At Scale, we believe investing in data is the key to unlocking success for eCommerce companies, and we&rsquo;re just getting started.&nbsp;
To get Early Access for Forge, <a href="https://scale.com/catalog" target="_blank" rel="noopener noreferrer">sign-up for the waitlist</a>.

Catalog Forge is the fastest way to create AI-generated product imagery in seconds. Forge is now in Early Access.

Announcing Scale Forge: Available for Early Access 

Becoming a veteran requires a transition from the regimented lifestyle as a service member into the civilian world. The majority of veterans <a href="https://www.cfr.org/backgrounder/demographics-us-military" target="_blank" rel="noopener noreferrer">joined out of high school or college</a>, and their adult lives have been shaped by their military service. We entrust our service members at a young age with responsibilities that they otherwise would never encounter in the civilian world. These jobs not only require a highly trained technical force, but are often life or death situations.
In my case, I was only 20 when I received orders to my ship. I was responsible for operating &amp; maintaining the computers and networks that powered the multi-million dollar AEGIS Combat system. By the time I was 22, I was on a 7 month deployment in the Persian Gulf, leading a team of fellow technicians supporting combat operations in Iraq &amp; Afghanistan during the 2011 withdrawal from Iraq.
<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fa540352ee602468a940cb12fc334d5e6">
My experience isn&rsquo;t unique. The average age of the US Military service member is<a href="https://download.militaryonesource.mil/12038/MOS/Reports/2018-demographics-report.pdf" target="_blank" rel="noopener noreferrer"> 28 years old</a>, with the vast majority of service members serving one tour and becoming a veteran before the age of 25. My fellow U.S. Navy Veteran at Scale, Nick Falconi, was only 25 when the Navy trusted him to oversee the operations of the nuclear reactor that powered the submarine he served onboard. He did this alone while in port as the Duty Officer, and acted as Senior Officer while the other Senior Officers slept at sea.
Serving in the military requires a commitment that is hard to describe. Having the privilege to do amazing things at such a young age makes it hard to define an identity outside of the military. Becoming a veteran means giving up the military which has defined your adult life up until that point. One day you are a fighter pilot, special forces, or intelligence analyst in the military, and the next you are just a person trying to figure out how to succeed in the civilian world.
It&rsquo;s already hard to give that up, and enter the civilian world, but it&rsquo;s even harder to translate that experience into a civilian career. After I left the Navy and transitioned, I made the decision to go to college back home in St. Louis, but I struggled being a 23 year old freshman in classes with 18 year old kids. After a year I decided to enter the workforce while still pursuing my degree. I had difficulty finding entry level IT positions, even though the AEGIS weapon system I supported was similar to the computers &amp; network equipment found at most companies. I use the same technical skills I did in the Navy daily in my current role as the Lead of the Infrastructure &amp; Security team for Scale&rsquo;s Federal Team.
It wasn&rsquo;t until I started getting involved in the St. Louis Veterans Community and volunteering that I found the mentors and resources that helped me succeed. I learned from other veterans in the St. Louis community like Steve Wahle &amp; Micheal Bamber how to use my military experience to my advantage. They introduced me to community leaders like Maxine Clark, the founder and former CEO of Build-A-Bear Workshops, who helped me land my first civilian job supporting the computers and networks that powered the Build-A-Bear retail empire.&nbsp;
I strongly believe a successful transition to civilian life is not only key to reintegrating veterans into the community, but it is also a good business decision. Veterans have deep technical knowledge &amp; leadership skills, along with a tenacity that is hard to train that makes successful employees who can jump right into tough problems. Scale recognizes this, and has been successful in its commitment to hire veterans. I&rsquo;m proud to say that Scale&rsquo;s veteran population is nearly 5% of the US based workforce exceeding the <a href="https://www.prweb.com/releases/new_research_from_findem_sheds_light_on_status_of_veterans_in_the_workforce/prweb17533903.htm" target="_blank" rel="noopener noreferrer">percentage of veterans in the tech workforce,</a> and one-third of veteran employees are recently separated.&nbsp;
I encourage everyone this Veteran&rsquo;s Day to commit to helping new veterans. If you are a transitioning service member looking for a new challenge &amp; mission, consider <a href="https://scale.com/blog" target="_blank" rel="noopener noreferrer">Scale.</a> Our mission is to accelerate the development of AI. If that sounds exciting to you, let's talk.

If you are a transitioning service member looking for a new challenge & mission, consider Scale. 

Scale is committed to hiring veterans

Veterans Day @ Scale

The 2022 TransformX conference may now be behind us, but you can still catch all of our sessions on-demand! Nearly 32,000 virtual and in-person attendees joined the 3-day event, which included almost 80 sessions and 108 speakers. 

TransformX 2022 Highlights and Recaps

Today we kicked off our third annual conference, TransformX, where the world's brightest AI leaders, visionaries, practitioners, and researchers engaged in conversations around the present and future of AI/ML.
At this year&rsquo;s event, we felt it was important to shine a light on how AI is going to define the next era of technology and power. Just like at the dawn of the internet, or cloud computing before it, the businesses that leverage AI will have an incredible competitive advantage to dominate their industries for years to come.&nbsp;
In fact, for many businesses, we&rsquo;re already at a nexus where it&rsquo;s AI or die. And that&rsquo;s because not all companies are equipped with the same resources to leverage the power of AI. Some organizations, often the biggest tech companies, are AI builders&mdash;they have the talent, resources, and experience to build their own, differentiated AI. But they often lack the high-quality data needed to feed their algorithms. On the other hand, there are organizations that aren&rsquo;t rooted in technology, but are prepared to leverage AI as a key asset and differentiator. They are looking to leverage this technology into their businesses without having to build their own ML team.
Whether an organization is already building AI or is at the starting line wanting to apply AI, it&rsquo;s clear that all organizations can reap tangible benefits from AI if they have the right tools and solutions tailored to their business. That&rsquo;s why today at TransformX, Scale introduced a suite of new products to unlock and operationalize AI for everyone &ndash; from startups to researchers and Fortune 500 companies to the U.S. government. These tools have already been battle tested in early access beta by companies such as <a href="https://opensea.io/" target="_blank" rel="noopener noreferrer">OpenSea</a>, <a href="https://www.voxelai.com/" target="_blank" rel="noopener noreferrer">Voxel</a> and <a href="https://picklerobot.com/" target="_blank" rel="noopener noreferrer">Pickle Robot</a>, and now they are fully available for everyone to adapt, develop and scale their use of AI.&nbsp;
<h2>Studio</h2>
Some labeling projects require in-depth subject-matter expertise. For example, medical initiatives have unique terminology and require deep domain knowledge. Studio provides organizations with the capability to run high quality, efficient labeling projects with their own team of experts. Features include the ability to pre-label datasets to reduce labeling time and auto-annotation to make it easy for anyone to label complex objects by simply drawing a box around the specific object. Studio supports all data types, including images, videos, audio, and text. Any team can quickly get started with Studio by signing up <a href="https://scale.com/studio" target="_blank" rel="noopener noreferrer">here</a>.&nbsp;
<h2>Nucleus Models&nbsp;</h2>
As many ML teams know, shipping a model isn&rsquo;t the end of the AI journey. To ensure a model performs well in all scenarios and avoids model failure in production, it is critical to test key scenarios beforehand. In 2020, we launched Nucleus, a new way to develop and test ML models. We are now improving Nucleus with new ML model testing functionalities that help catch regression over time and help teams to choose the best model through efficient model comparisons and scenario-level insights. Learn more about Nucleus and its model features <a href="https://scale.com/nucleus" target="_blank" rel="noopener noreferrer">here</a>.
<h2>Content Understanding AI</h2>
One of the biggest challenges for companies that invest in user-generated content is to recommend relevant content that keeps users engaged. With Content Understanding AI, product teams can easily enrich user-generated content to customize and iterate their content&rsquo;s taxonomy - ultimately increasing engagement. Data enrichment also gives product teams insights into their user&rsquo;s content, such as sentiment and mood, to help identify growth areas. Additionally, trend detection leverages ML techniques like clustering, topic aggregation and extraction, as well as Scale human-in-the-loop features to detect and understand new content trends surfacing on platforms. Learn more about Scale&rsquo;s Content Understanding <a href="https://scale.com/content-understanding" target="_blank" rel="noopener noreferrer">here</a>.&nbsp;
<h2>Document AI Go&nbsp;</h2>
In 2020, we launched Document AI to process complex documents up to near perfect accuracy in seconds. This year, we launched Document AI Go. Document AI Go is a self-serve machine learning platform that enables companies to quickly classify and extract data from their documents in just a matter of seconds. Unlike competing products, Document AI and Document AI Go are powered by foundation models and don&rsquo;t rely on templates, so they can handle the most complex and unstructured documents. Any organization can quickly sign up and start uploading documents <a href="https://scale.com/document-ai#pricing" target="_blank" rel="noopener noreferrer">here</a>.
We&rsquo;re thrilled to introduce these tools and put them in the hands of every company. We can only imagine what&rsquo;s possible when every company can access the best quality AI tools and tailored models. I believe that by making it easier for more companies to unlock and operationalize AI, we will experience a world where AI meaningfully supports humans as they solve the world&rsquo;s biggest problems.

At TransformX this year, we're sharing details on Studio, Nucleus Models, Content Understanding AI, and Document AI Go.

TransformX 2022: Operationalizing AI

Last week, and every October, we celebrated National Coming Out Day. The celebration started in 1988 on the first anniversary of the second National March on Washington for Lesbian and Gay Rights in October 1987.&nbsp;
<a href="https://lgbtq.gmu.edu/programs-services/national-coming-out-day/" target="_blank" rel="noopener noreferrer">The march</a>'s goal was to draw attention to the federal government&rsquo;s inaction in confronting the AIDS crisis and the Supreme Court&rsquo;s 1986 ruling upholding Georgia&rsquo;s anti-gay law. According to NCOD (National Coming Out Day), protesters flooded D.C. over a five day period and demanded legal recognition for gay and lesbian couples and the funding for AIDS research. The march also marked<a href="https://19thnews.org/2021/10/the-history-of-national-coming-out-day-contains-both-pride-and-pain/" target="_blank" rel="noopener noreferrer"> the unveiling</a> of the AIDS memorial quilt, a massive patchwork honoring those lost to the virus, and at the time an unprecedented show of support for gay rights.
National Coming Out Day is meant to be a celebration of individuals within the LGBTQA+ community who openly identify as lesbian, gay, bisexual, transgender, non-binary, queer, asexual or intersex. It&rsquo;s a day for honoring the act of coming out, and all the hopes, fears, dreams and expectations for the future.
Whether it is at home, work, or in your community, coming out is the first step to living authentically. Scale is committed to creating a culture of inclusion that supports all Scaliens &ndash; no matter how they identify.
We want to express gratitude to the Scaliens who have opened up to share their stories for this year&rsquo;s National Coming Out Day:
<h2>Abe</h2>
I grew up in a town of 1300 people in the middle of Southern Illinois. Where the pressures of never coming out and the expectations of settling down were standard.
We moved from Missouri to Illinois around that time to help my grandma after my grandpa had passed suddenly of cancer.&nbsp;She lived on a farm in the middle of nowhere and that&rsquo;s where me and my twin brother, along with an older brother and sister stayed for a few years until we moved to another house in the country when I was 9. During my tenure in grade school, there were a lot of slurs thrown around by kids who didn&rsquo;t really understand what they meant.&nbsp;I never used them personally, because that's not who I&rsquo;ve ever been. But I know if one of those slurs was directed at someone, it was a really bad thing and meant that kid was now viewed as less than a human.&nbsp;So I tried to fly under the radar.&nbsp;
In middle school, I tried dating girls, because that was what you were supposed to do: date women, get married and have kids. I even pined after my best friend since the 4th grade until we were out of high school, but she never really returned the way I felt.&nbsp;
When I got to high school, a kid in the grade above me was the first openly gay person I&rsquo;d ever met.&nbsp;I admired the heck out of him for doing something I thought I could never do, and also didn&rsquo;t understand why he&rsquo;d want to subject himself to the small-minded mentality of the town we had grown up in.&nbsp;&nbsp;
I finally graduated high school in 2000 and went to a small community college over by Centralia, IL.&nbsp;While going to college there, I was working at a gas station and an attractive male customer came in&hellip;that&rsquo;s when I had my &ldquo;holy CRAP, I&rsquo;m gay&rdquo; moment. I think many of you know what I&rsquo;m talking about: the time you finally admit to yourself that you&rsquo;re definitely not straight.&nbsp;
I met the person who would become my future wife on the first day of freshman orientation.&nbsp;We were inseparable. We did everything together, enjoyed all the same activities like driving around, singing songs on the radio, and just enjoyed being around each other.&nbsp;After a few years, I thought to myself, &ldquo;If there was anyone you could make a straight life with, it would be this woman.&rdquo; So, I did what anybody who came from a town of 1,300 people would do: I asked her to marry me, driving down IL-4 one night.&nbsp;I was between Stelleville and Sparta when I popped the question&hellip;while driving&hellip;and didn&rsquo;t have a ring. We were married on November 6th, 2004 at what was supposed to be a beautiful ceremony outside her parents house.
Even before I had admitted to myself who I truly was, I had explored being with men.&nbsp;With that being said, a few months after I was married, I met another known gay person from my hometown (who had graduated high school right before I had entered) and struck up a friendship with him.&nbsp;Problem was, I ended up kissing him after the friendship started.&nbsp;Now this was the first time I had ever kissed another man, and my whole life changed.&nbsp;I couldn&rsquo;t eat and seriously lost 40 pounds in 2 weeks.&nbsp;I couldn&rsquo;t sleep, because I just thought I was always going to be married to my wife and now that I had kissed another man, I knew I couldn&rsquo;t go on living as a straight man.
So one night, I came home after a late shift, woke my wife up and told her the news, &ldquo;Honey, I don&rsquo;t know how to tell you this, but I&rsquo;m gay.&nbsp;Here&rsquo;s how it happened [see story above] and I&rsquo;m truly sorry.&nbsp;I thought I could be straight and I just can&rsquo;t be anymore.&rdquo;&nbsp;There was a lot of crying that night between the 2 of us, a lot of discussion about the future and what it looked like for us, and how to start tomorrow off. I ended up coming out to everyone else that was important around me for the next few weeks and really received love and support from 99% of my family unit (I&rsquo;ve always had the philosophy that family is the ones you choose as your family, so that doesn&rsquo;t just mean people I&rsquo;m related to).&nbsp;We ended up selling the house and 1 year and 1 month after we were married we officially signed the divorce papers.&nbsp;
I still look back on that night with awe on how I ever had the courage to do it.&nbsp;I know I hurt some people, especially my ex-wife, but we still talk to this day and joke about that time in our life. She&rsquo;s remarried with a beautiful daughter and I&rsquo;ve even met her new husband.&nbsp;I&rsquo;ve since moved to St. Louis where I try to be a beacon in the community, including heading volunteer efforts for an LGBTQ+ group I&rsquo;m a member of, recently chairing a committee that helped raise $1500 for The Metro Trans Umbrella Group (MTUG) in one night.&nbsp;
Yes, it was hard to come out, but I never liked closets anyway.&nbsp;The sunshine is so much better outside of them.
<h2>Willow</h2>
I knew from an early age that I was gay and transgender, but it took me many years to come out. The social stigma felt very high. Gayness was used as an insult, and transgender people were a bad punchline at best.
It took me 14 years of growing up to decide that I didn&rsquo;t really care about the social stigma. During that time, I had focused on becoming an engineer and a soldier, stereotypically male professions. While deployed in Afghanistan, I was struggling to understand why I felt more depressed about the thought of returning home than staying in theater, and I realized that I was living my life inauthentically in a way that was leaving me devastatingly unhappy.
When I got back home, I came out as queer and made plans to leave the military. In 2014, transgender people were still barred for military service, and while it seemed like that might change, I figured it would be a messy policy process with too much back and forth for me to want to live through (unfortunately this turned out to be correct).&nbsp;The first person I came out to as transgender was my long term girlfriend, who was incredibly supportive and caring. Next, I came out to my parents who said they loved and supported me and promised to assist in any way they could with transition. My immediate supervisor was also supportive, which made managing my exit from the military much easier and simpler. Even the most supportive coming out was very hard for me psychologically: it felt like a terrible admission of fragility and stigma. How much more difficult for those without support?
I left the military, went back to school, and changed careers and genders. It was very challenging emotionally, but bluntly far easier for me than most other transgender people as a result of relative socioeconomic privilege and education, and due to the supportive friends and family around me. For this reason, it has always been important to me to be out as queer and transgender in my social life and business life. I do not treat these aspects of who I am as secrets, and I hope that being out makes coming out less of a burden for other people over time.
<h2>Richard&nbsp;</h2>
I knew I was gay years before I met an openly gay person. I was 11 years old and watching TV when all of a sudden I found myself attracted to a male actor. For whatever reason, in that moment, I had no doubt that I was gay. I didn&rsquo;t want to be - &ldquo;gay&rdquo; was a prejorative even among 5th graders in midwest suburbia - but it was a fundamental truth, as undeniable as 2+2=4.
It never occurred to me that I could hide this forever. Living in the closet requires telling dozens of lies daily. Some are overt. Many are subtle, a slight blur of the truth. It&rsquo;s an exhausting existence, one that I personally was not capable of. Rationally, I knew that I had nothing to be ashamed of. But as much as that made logical sense, it was still emotionally terrifying - would I lose my friends? Could I have kids and build a family? Could I have a &ldquo;respectable&rdquo; career and be a &ldquo;respectable&rdquo; member of society?&nbsp;
I&rsquo;m a big believer in &ldquo;getting your reps in&rdquo; - you need to practice in order to build muscles, whether they&rsquo;re physical or emotional. Coming out isn&rsquo;t a one-time thing. You come out over and over again. To your friends&rsquo; parents. To new colleagues. To old friends you didn&rsquo;t really keep in touch with, and you aren&rsquo;t quite sure if the gossip reached them. It took me a few years to learn how to come out without showing my fear externally, and a few more to not feel it internally either. It took dozens of reps to rewire the part of my brain that would go into overdrive - that flight or fight response that kicks in when you subconsciously perceive that you&rsquo;re in danger.
As a teenager, coming out was the hardest thing I had ever done in my life. Arguably, it still is. There were no openly gay students at my high school. There were no openly gay men in my college fraternity. It was terrifying, but what was once a source of terror is now a source of pride.
One of my fondest memories was visiting my brother in 2012. I was 20 years old, and he had joined some small startup called Dropbox in some far away place called San Francisco. I was eating lunch served by company chefs in the office cafeteria - that in itself was a novel concept - and overheard a same-sex couple, surrounded by friends and colleagues, talking about their baby. My mind was blown by how at ease they were, and how &ldquo;normal&rdquo; it all felt. At that moment, I really felt that I could both be gay and have the foundational things I wanted - friends, family, and a career. My dream was a possibility, and it was up to me to make it a reality. For this, I will always be grateful.
During my lifetime, the world has overall become a more accepting place. I&rsquo;m optimistic that, even if there are some bumps in the road, we&rsquo;ll continue to make progress, and more and more people will have the freedom to live authentically.&nbsp;

The celebration started in 1988 on the first anniversary of the second National March on Washington for Lesbian and Gay Rights in October 1987. 

National Coming Out Day is a celebration of individuals within the LGBTQA+ community 

Scale Out: National Coming Out Day 

Model evaluation is one of the most important prerequisites prior to shipping an ML model. If you don’t properly evaluate your model, performance regressions might appear in production and have a direct and negative impact on your business. For example, if a faulty pedestrian detection model is deployed onto an autonomous vehicle, it can potentially cause an accident. In its most basic form, model evaluation is very straightforward, but as we discussed in our <a href="https://scale.com/blog/find-best-model" rel="noopener noreferrer" target="_blank">previous blog</a>, approaches that are too simple aren’t really reliable. Therefore, most ML teams aspire to maintain a mature model evaluation pipeline in order to systematically understand model performance. A mature evaluation pipeline covers the entire lifecycle of ML development:<ol><li>Model training and re-training</li><li>Model testing and validation</li><li>Model performance monitoring in production&nbsp;&nbsp;</li></ol>To cover all these aspects, ML teams need sophisticated tooling, which is why we assembled a list of 4 MLOps tools that we believe will help you build a great model evaluation pipeline:Weights &amp; Biases<img src="https://lh5.googleusercontent.com/FT9Rlbx977RDuvofnHrtTOxQPjRAjHtncmZQJZel9sDeCRiRYgdle8ubOonzVkVGgkxhxuv_7J5oSyXSSMpU2yZa-Vkv4zYxXibg2Z-nB42bjTd3hl4O8PaZc_Vpluk7S8Wsw5M8GpxNOw4mep2TOiqYX08B0haUhu-_YqmI9YXes0WfSIZAJV1KGg"><a href="https://wandb.ai/site/experiment-tracking" rel="noopener noreferrer" target="_blank">Source</a>Weights &amp; Biases (W&amp;B, for short) is a great tool for logging information and analyzing performance during model training. It is designed to track standard training metrics like loss, training dataset (“train”) accuracy, validation (“val”) dataset accuracy, GPU utilization, and more.&nbsp; W&amp;B will also log your raw data like images or point clouds along with their matching predictions like bounding boxes, cuboids, and segmentation masks. Moreover, W&amp;B supports experiment tracking. Users can log training run configurations, including sets of hyperparameters, and use these to reproduce the experiment later. Already widely adopted in the ML community, W&amp;B is a more powerful cloud-hosted alternative to TensorBoard, especially if teams want to converge on a collaborative tool.COCO Eval API<img src="https://lh5.googleusercontent.com/SXczlMWaBJoU7IofJKyyrJoP1yYnfo_m7VvAY55w_eZMU-tvP9Bz5Q69U87f18Qv_6gvZ7FGWLjdnH2PZqfJm1bp9BxwyqQvuas3FT-hxgZkBwExjg_cifCN8noswnenhVnDsPjo1P8inay5wUrFAMHhFhvTNiCaITChXgcP8-5DCF-a-g036X6mjA">While there are a lot of tools available to log training and evaluation metrics, COCO’s eval API is among the handful that provide these metric definitions and implementation out of the box. Released as a toolkit for interacting with the MSCOCO dataset, the COCO eval API has now become the standard tool for many ML developers working on computer vision use cases like object detection. The API supports the calculation of IoU, precision, and recall on bounding boxes, segmentation masks, and keypoints. It also allows users to aggregate these scores—both per item and per class. Although very bare bones, the COCO eval API is a handy tool for obtaining standard metrics.Scale Validate<img src="https://lh6.googleusercontent.com/H5OAAYhwy544Pj2iFx8yuDCeKxPJ7phxvazeRPgDgYI5s8tUTHra66cuqLzgxHTcCT6qOJuKG8Wd3DQTnkP754VhCekZMV2wqQXZxgUfp2oTWNIhwnGtPIp4a4g5gUphE41hhAJ_X3DaJU-hx7bNbJ9I3LfALBWJg7nrFGslKQkcN0-i5G11EjvPsA">While metric computation and logging are essential, they are not sufficient to uncover the true performance of the model prior to deployment for reasons such as data inequality and comparison complexities (<a href="https://scale.com/blog/find-best-model" rel="noopener noreferrer" target="_blank">you can learn more here</a>). ML teams frequently tackle these challenges using Scale Validate, which provides a structured framework for testing and comparing ML models. Validate allows you to:<ol><li>Compare models on a rich set of pre-computed metrics e.g. precision, recall, confusion matrix, IoU or mAP.</li><li>Find underperforming slices through automatic clustering based on image embeddings and metric scores&nbsp;&nbsp;</li><li>Set up tests on mission-critical scenarios to track progress on metrics of choice over time and catch regressions.</li></ol>Through these strategies, Validate helps dissect obscure aggregate metrics into more granular insights and ensures that the model being shipped performs better not just on the aggregate but also on the more critical and rare long-tail situations.&nbsp;Why Labs<img src="https://lh5.googleusercontent.com/AO7znH2bIj4K__AVI3X-NGBdBqbpMhWXE_tUQ3Q-3XM_n4Xybl-1DG7vsn2BTWbsPJpn3LqQgMif9Smq-GOsl3tD5HyZjRmurRj8JgErcIVp2W1IC_ZBO54wyiJfPhg8rJfzsAaZG18PUXy7XEmDOtn-7W7lWp-XXQSm7nFzYzeDjlV7zOtwYva1PA">After a model has been deployed to production, it is crucial to keep monitoring its performance to ensure that the model performs as expected. WhyLabs is an observability platform designed to monitor ML model performance degradation. Users can create monitors to target a particular performance metric—typically, precision, recall or F1 score. Once the monitors are set, WhyLabs tracks the metrics relative to baselines set by the user. These baselines can be based on a dynamic range, like an average metric value over a historic time interval or a static fixed value. Using these monitors, users can also set corresponding actions if the value crosses certain thresholds, like alert messages. WhyLabs can be deployed in production cloud environments including AWS and GCP, or on-premises.&nbsp;Great AI applications require dependable ML models and an ML model’s dependability can’t be ascertained without foolproof evaluation. Using these tools, ML teams can build a reliable evaluation pipeline to uncover true model performance and avoid any regressions in production. <a href="https://dashboard.scale.com/nucleus/scenario-tests?utm_source=blogpost&amp;utm_medium=website&amp;utm_campaign=top_eval_tools" rel="noopener noreferrer" target="_blank">Get started with Scale Validate</a> today to uncover and fix regressions over critical scenarios.

Model evaluation is one of the most important prerequisites prior to shipping an ML model.

Top 4 Tools to Build Your ML Model Evaluation Pipeline

URL redirections for Website. Disclaimer: takes 5~ min for each added or edited.

url-redirects

Use relative path. use `/blog/foo` not `https://scale.com/blog/foo`

blog-filter-type

homepage-quotes

logos

homepage-content

quotes

blog-category-model

announcement-banner

homepage-resources

homepage-event

custom-fonts

author

Event model for new marketing events page

event

Optional second image intended to show on the grid view, should be a more squared image closer to 1:1 aspect ratio than the one used in the Cover field.

guides

announcement-news

homepage-research

legal

slugs must be all lowercase. ie: ‘my-slug’

eval

datastream

summit-2023-speakers

event-model

Toggle if the event is featured. Featured events will be moved to the top of the list

customers

Display order on the page /customers. Higher values show first

summit-2023-agenda

summit-2023-partners

enterprise-use-cases-categories

enterprise-use-cases-categories-items

partners

data-samples

enterprise-use-cases

Use only one image or video option. Add another item to add another source.

collapsible-sections

blog-model

Option to list or not list this post in the Blog index. Enabled = not list in the index, Disabled = list in the index

research

basecamp-speakers

Example: https://www.twitter.com/username

Example: https://www.linekin.com/in/username

basecamp-speakers-events

basecamp-agenda

leaderboards

Text for a short description like: Deprecated (as of January 2025)

Keyword for search. Add multiple keyword with comma. Example: hle, hleto, vista, enigma

Products

Enterprise

Government

Resources

Customers

Leaderboards →

Blog