Overview

We strive to bring Artificial Intelligence features to Nextcloud. This section highlights these features, how they work and where to find them. All of these features are completely optional. If you want to have them on your server, you need install them via separate Nextcloud Apps.

Overview of AI features

Nextcloud uses modularity to separate raw AI functionality from the Graphical User interfaces and apps that make use of said functionality. Each instance can thus make use of various backends that provide the functionality for the same frontends and the same functionality can be implemented by multiple apps using on-premises processing or third-party AI service providers.

Feature

App

Rating

Open source

Freely available model

Freely available training data

Privacy: Keeps data on premises

Smart inbox

Mail

Green

Yes

Yes

Yes

Yes

Image object recognition

Recognize

Green

Yes

Yes

Yes

Yes

Image face recognition

Recognize

Green

Yes

Yes

Yes

Yes

Video action recognition

Recognize

Green

Yes

Yes

Yes

Yes

Audio music genre recognition

Recognize

Green

Yes

Yes

Yes

Yes

Suspicious login detection

Suspicious Login

Green

Yes

Yes

Yes

Yes

Related resources

Related Resources

Green

Yes

Yes

Yes

Yes

Recommended files

recommended_files

Green

Yes

Yes

Yes

Yes

Text processing using LLMs

llm2 (ExApp)

Green

Yes

Yes - Llama 3.1 model by Meta

Yes

Yes

OpenAI and LocalAI integration (via OpenAI API)

Red

No

No

No

No

OpenAI and LocalAI integration (via LocalAI)

Yellow

Yes

Yes - e.g. Llama models by Meta

No

Yes

Machine translation

Local Machine Translation 2 (ExApp)

Green

Yes

Yes - MADLAD models by Google

Yes

Yes

DeepL integration

Red

No

No

No

No

OpenAI and LocalAI integration (via OpenAI API)

Red

No

No

No

No

OpenAI and LocalAI integration (via LocalAI)

Green

Yes

Yes

Yes

Yes

Speech to Text

Local Whisper Speech-To-Text 2 (ExApp)

Yellow

Yes

Yes - Whisper models by OpenAI

No

Yes

OpenAI and LocalAI integration

Yellow

Yes

Yes - Whisper models by OpenAI

No

No

Replicate integration

Yellow

Yes

Yes - Whisper models by OpenAI

No

No

Image generation

Local Stable Diffusion

Yellow

Yes

Yes - StableDiffusion XL model by StabilityAI

No

Yes

OpenAI and LocalAI integration (via OpenAI API)

Red

No

No

No

No

OpenAI and LocalAI integration (via LocalAI)

Yellow

Yes

Yes - StableDiffusion models by StabilityAI

No

Yes

Replicate integration

Yellow

Yes

Yes - StableDiffusion models by StabilityAI

No

No

Local large language model 2 (ExApp)

Yellow

Yes

Yes

No

Yes

OpenAI and LocalAI integration (via OpenAI API)

Red

No

No

No

No

OpenAI and LocalAI integration (via LocalAI)

Green

Yes

Yes

Yes

Yes

Context Chat

Nextcloud Assistant Context Chat

Yellow

Yes

Yes

No

Yes

Nextcloud Assistant Context Chat (Backend)

Yellow

Yes

Yes

No

Yes

Ethical AI Rating

Until Hub 3, we succeeded in offering features without relying on proprietary blobs or third party services. Yet, while there is a large community developing ethical, safe and privacy-respecting technologies, there are many other relevant technologies users might want to use. We want to provide users with these cutting-edge technologies – but also be transparent. For some use cases, ChatGPT might be a reasonable solution, while for more private, professional or sensitive data, it is paramount to have a local, on-prem, open solution. To differentiate these, we developed an Ethical AI Rating.

The rating has four levels:
  • Red

  • Orange

  • Yellow

  • Green

It is based on points from these factors:
  • Is the software (both for inferencing and training) under a free and open source license?

  • Is the trained model freely available for self-hosting?

  • Is the training data available and free to use?

If all of these points are met, we give a Green label. If none are met, it is Red. If 1 condition is met, it is Orange and if 2 conditions are met, Yellow.

Features used by other apps

Some of our AI features are realized as generic APIs that any app can use and any app can provide an implementation for by registering a provider. So far, these are Machine translation, Speech-To-Text, Image generation, Text processing and Context Chat.

Text processing

As you can see in the table above we have multiple apps offering text processing using Large language models. In downstream apps like Context Chat and assistant, users can use the text processing functionality regardless of which app implements it behind the scenes.

Frontend apps

  • Text for offering an inline graphical UI for the various tasks

  • Assistant for offering a graphical UI for the various tasks and a smart picker

  • Mail for summarizing mail threads (see the Nextcloud Mail docs for how to enable this)

  • Summary Bot for summarizing chat histories in Talk

Backend apps

Machine translation

As you can see in the table above we have multiple apps offering machine translation capabilities. Each app brings its own set of supported languages. In downstream apps like the Text app, users can use the translation functionality regardless of which app implements it behind the scenes.

Frontend apps

  • Text for offering the translation menu

  • Assistant offering a graphical translation UI

  • Analytics for translating graph labels

Backend apps

  • translate2 (ExApp) - Runs open source AI translation models locally on your own server hardware (Customer support available upon request)

  • integration_deepl - Integrates with the deepl API to provide translation functionality from Deepl.com servers (Only community supported)

Speech-To-Text

As you can see in the table above we have multiple apps offering Speech-To-Text capabilities. In downstream apps like the Talk app, users can use the transcription functionality regardless of which app implements it behind the scenes.

Frontend apps

Backend apps

Image generation

As you can see in the table above we have multiple apps offering Image generation capabilities. In downstream apps like the Text-to-Image helper app, users can use the image generation functionality regardless of which app implements it behind the scenes.

Frontend apps

  • Assistant for offering a graphical UI and a smart picker

Backend apps

  • text2image_stablediffusion (Customer support available upon request)

  • OpenAI and LocalAI integration (via OpenAI API) - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see AI as a Service)

  • integration_replicate - Integrates with the replicate API to provide AI functionality from replicate servers (see AI as a Service)

Context Chat

Our Context Chat feature was introduced in Nextcloud Hub 7 (v28). It allows asking questions to the assistant related to your documents in Nextcloud. You will need to install both the context_chat app as well as the context_chat_backend External App. Be prepared that things might break or be a little rough around the edges. We look forward to your feedback!

Frontend apps

  • Assistant for offering a graphical UI for the context chat tasks

Backend apps

Provider apps

Apps can integrate their content with Context Chat to make it available for querying using Context Chat. The following apps have implemented this integration so far:

Frequently Asked Questions

Why is my prompt slow?

Reasons for slow performance from a user perspective can be

  • Using CPU processing instead of GPU (sometimes this limit is imposed by the used app)

  • High user demand for the feature: User prompts and AI tasks are usually processed in the order they are received, which can cause delays when a lot of users access these features at the same time.