This post was co-authored with Max Silfverberg (Data Scientist, AI Solutions Lead), Antti Hallavo (Lead AI Software Engineer), and Pontus Huotari (Lead Data Scientist). We work at Alma Media, a Finnish digital services, marketplaces, and media company. One of our focus areas is developing AI/ML solutions for real estate listing services, where understanding image content plays a crucial role.
Alma Media’s real estate services process hundreds of thousands of listings annually. Most of these listings come with dozens of photos, but no metadata describing what they show. Meanwhile, search, recommendations, and a range of internal use cases all benefit from knowing whether a photo depicts a kitchen, a floor plan, or a garden.
Our solution is to automatically tag photos with room-type and content classes. Our room types include LIVING ROOM, KITCHEN, and BEDROOM. We also tag schematic content like floor plans and site plans. Additionally, we recognize realtor marketing materials, aerial shots, and garden photos. Altogether, there are 23 classes. As Figure 1 illustrates, this is a classic multi-label classification task; the same space can encompass several room types at once.
Figure 1. Our system should tag this photo as LIVING ROOM and STAIRCASE. The dining room visible through a doorway should not affect the class. Photo by Clay Banks on Unsplash.
At first glance, this may seem straightforward, but we face several tricky decisions. How should we handle a living room photo that shows a bedroom through a doorway? What if a photo shows only 10% living room and the remaining 90% is dining area? The answers depend entirely on the application.
If we need to find all photos showing kitchens, we also want to identify living room photos that happen to feature a kitchen in the background. However, if a user specifically requests kitchen photos, we should only return images where the kitchen is the focal point. Classification confidence is key to making these distinctions, but depending on how you design your classifier, you might not have access to that information.
Image classifiers can be built in several ways. The modern default is to use a third-party API that internally leverages a vision-language model (VLM) to analyze images and generate tags based on a prompt. Another option is to train image classifiers on top of open-source vision transformer (ViT) foundation models like Google SigLIP or Meta DINO, either freezing the foundation model or fine-tuning it on your data. Each approach comes with its own advantages and trade-offs.
There is already a wealth of research comparing these approaches based on numerical performance [1]. This post goes further by addressing a frequently overlooked question: How should you build image classifiers in a business context?
Three Questions to Ask Before Training Anything
We built our proprietary classifiers by fine-tuning google/siglip-base-patch16-224. But why did we choose this path? Check Figure 2 for a quick summary, or read on for the full story.
Figure 2. Should you prompt an API or train your own classifier, with or without fine-tuning? Image by author.
Question 1: Prompt an API or Train Your Own Model?
The choice between prompting through an external API and building your own classifier depends heavily on your specific use case. First, consider whether your classification task can even be prompted. It’s easy to prompt a car classifier or a kitchen appliance classifier, but what about more ambiguous categories like 'cozy living room' or 'modern kitchen'? In our experience, tasks that require fine-grained, domain-specific distinctions—such as distinguishing between a floor plan and a site plan—are difficult to handle with generic prompts. An external API might also lag in recognizing niche classes or adapting to new ones.
Additionally, think about data privacy and control. If your data includes sensitive information, or if you need to comply with strict regulations, keeping the classification process in-house may be a priority. APIs can also introduce latency and per-request costs, which become significant at the scale of hundreds of thousands of listings. On the other hand, training your own model requires significant upfront investment in data labeling, compute, and maintenance.
Question 2: Use a Pre-Trained Model or Fine-Tune It?
If you decide to train your own classifier, the next decision is whether to use a frozen pre-trained model or fine-tune it. A frozen model acts as a feature extractor; you only train a lightweight classifier head on top of the extracted features. This is faster and requires less labeled data, but the features may not be optimal for your specific domain. In our case, real estate photos are quite distinct—they often feature wide-angle lenses, specific lighting, and furniture styles. Fine-tuning the entire model, or at least the later layers, allows the model to adapt these features to our data, leading to better accuracy.
However, fine-tuning is not without risks. It requires more labeled data and computational resources, and there is a risk of overfitting, especially if your dataset is small or biased. Fine-tuning also means you need to manage version control and model updates carefully to ensure consistent performance over time.
Question 3: What Are the Real-World Constraints?
Beyond technical performance, we have to consider operational constraints. Our team needed a solution that could be integrated into our existing infrastructure with minimal disruption. An API might be simpler to deploy, but it creates a dependency on an external provider. If they change their model or pricing, we could be forced to adapt quickly. By fine-tuning our own model, we maintain control over the entire pipeline, from deployment to monitoring.
Another factor is explainability. In a business context, we often need to justify our model’s decisions to stakeholders or clients. With a custom model, we can analyze attention maps and feature importance to understand why a particular photo is tagged as a kitchen, which is harder to do with a black-box API.
Conclusion: Fine-Tuning Was Right for Us, But Is It Right for You?
In our case, fine-tuning SigLIP was the right choice because we had a large, labeled dataset of real estate photos, strict data privacy requirements, and a need for precise, domain-specific classes. The performance gains and control outweighed the costs of training and maintenance.
However, this is not always the right call. If your task is generic, your data is limited, or your team lacks the resources to manage a custom model, prompting an API or using a frozen pre-trained model might be more practical. In 2026, we expect to see even more powerful foundation models and more accessible fine-tuning tools, which could shift the balance further toward custom solutions. For now, we recommend asking the three questions above before making your decision.
This post was originally published on the Alma Media engineering blog.