Apple Intelligence is the name of Apple’s effort for Artificial Intelligence. The company says it “draws on your personal context while setting a brand-new standard for privacy in AI.”
It was introduced during the WWDC 2024 keynote, and it will be a central part of Apple’s iPhone, iPad, and Mac devices, starting with iOS 18, iPadOS 18, and macOS Sequoia.
Features
These are some of the Apple Intelligence features we’ll see on iPhone, iPad, and Mac:
- Writing Tools: Users can rewrite, proofread, and summarize text nearly everywhere they write, including Mail, Notes, Pages, and third-party apps;
- Image Playground: Users can create playful images in seconds, choosing from Animation, Illustration, or Sketch. This app is built right into apps like Messages and is also available in a dedicated app;
- Memories in Photos: Users can create stories they want to see just by typing a description. Apple Intelligence will pick out the best photos and videos based on the description, craft a storyline with chapters based on themes identified from the photos, and arrange them into a movie with its own narrative arc;
- Clean Up tool: This Photos app feature can identify and remove distracting objects in the background of a photo without accidentally altering the subject;
- Siri: Users type to Siri and switch between text and voice to communicate with Siri in whatever way feels right for the moment.
- ChatGPT integration: When you feel Apple Intelligence isn’t enough, you can allow ChatGPT to access Writing Tools and other features for a better response.
Tim Cook explains Apple and OpenAI’s ChatGPT partnership
Rumors were true, and Apple has partnered with OpenAI. According to the company, these two projects work seamlessly, but they have core features that separate them.
With Apple AI, the company ensures that all data is private through Private Cloud Compute, while OpenAI’s ChatGPT usually collects user data. In an interview with YouTuber Marques Brownlee, Apple’s CEO Tim Cook explained the core difference between Apple Intelligence and ChatGPT partnership.
“There’s Private Cloud Computing, and there’s the arrangement with OpenAI,” says Tim Cook. “These two things are different. So, if you look at Private Cloud Compute, we’re utilizing the same basic architecture as the silicon that’s in the iPhone 15. We’re using the same software, and so we believe that we’ve done it in such a way that it’s as safe and secure and private in the Private Cloud Compute as in the device.”
That means Apple won’t collect user’s data, won’t make a profile of the user, or take this data to sell it elsewhere. Cupertino aimed to extend the iPhone’s on-device processing to the next level with a level of security that people are used to with their iPhones.
Tim Cook continues: “So we really, we really worked on this on a lot and put a lot of work behind that arrow to be sure that if you’re working on something that requires world knowledge, so you’re out of the domain of personal context and so forth, then you may want to go and use one of the large language models that are on the market, and we will be selected what we feel is the best one with OpenAI and ChatGPT.”
That said, all personal requests related to Apple’s built-in apps, such as Messages, Mail, Calendar, and more, will use the company’s intelligence. In contrast, “world knowledge” can be requested for OpenAI ChatGPT and later for other large language models.
New LLMs can join the party later
While Apple will first integrate with OpenAI, the company plans to work with other LLms as well. For example, Cupertino is in talks with Google to license Gemini.
A report also claims Apple will use Baidu for its generative AI functions in China. Baidu’s Ernie Bot is a ChatGPT rival and one of the more than 40 AI models from China that local regulators have approved. A partnership with Apple would be a big win for Baidu, considering the growing competition in the region.
Release date
Apple Intelligence is expected to debut with iOS 18 public beta. Although it’s unclear if the new AI features are going to be available with the first public beta version, which should be released in the coming days, Apple says users will be able to test these functions before an official iOS 18 version is released to everyone.
On July 15, Cupertino re-released iOS 18 beta 3, iPadOS 18 beta 3, and macOS Sequoia beta 3, which means both public betas and Apple Intelligence are coming soon. That said, it’s important to note that Apple Intelligence will launch in beta. It’s unclear when the final version is going to be released.
Apple Intelligence compatible devices
During the WWDC 2024 keynote, Apple announced which devices will be compatible with its Intelligence:
- iPhone 15 Pro models, and possibly all iPhone 16 models
- M1 iPad models or newer (such as the M4 iPad Pro)
- Apple Silicon Macs running macOS Sequoia
Apple papers suggest where its AI efforts are at
AI model for instruction-based image editing
In February, Apple released a revolutionary AI model for instruction-based image editing. According to a paper published by Apple researchers, instruction-based image editing improves the controllability and flexibility of image manipulation via natural commands without elaborate descriptions or regional masks. The study shows “promising capabilities in cross-modal understanding and visual-aware response generation via LM” as they investigated how MLLMs facilitate edit instructions and MLLM-guided image editing.
This image editing AI model made by Apple can produce concise and clear instructions for the editing process, create Photoshop-style modifications, optimize photo quality, and edit specific elements of a picture, such as faces, eyes, hair, clothes, and accessories.
MM1: Apple’s AI model
In March, Apple researchers published a paper highlighting how they’re training a new large language model (LLM).
Called MM1, this LLM can integrate text and visual information simultaneously. The paper offers an interesting look at the importance of various architectural components and data choices. The researchers say they were able to “demonstrate that for large-scale multimodal pre-training using a careful mix of image-caption, interleaved image-text, and text-only data is crucial for achieving state-of-the-art (SOTA) few-shot results across multiple benchmarks, compared to other published pre-training results.”
In addition, they showed that “the image encoder together with image resolution and the image token count has a substantial impact, while the vision-language connector design is of comparatively negligible importance.”
Apple’s MM1 AI model uses a family of multimodal models with up to 30 billion parameters, consisting of both dense models and mixture-of-experts (MoE) variants, that are state-of-the-art in pre-training metrics and achieve competitive performance after supervised fine-tuning on a range of established multimodal benchmarks.
ReALM could be better than OpenAI’s GPT-4
Apple researchers have published a paper about a new AI model. According to the company, ReALM is a language model that can understand and successfully handle contexts of different kinds. With that, users can ask about something on the screen or run in the background, and the language model can still understand the context and give the proper answer.
This is the third paper regarding AI that Apple has published in the past few months. These studies only tease the upcoming AI features of iOS 18, macOS 15, and Apple’s newest operating systems. In the paper, Apple researchers say, “Reference resolution is an important problem, one that is essential to understand and successfully handle context of different kinds.
One example is a user asking for pharmacies near them. After a list is presented, something Siri could do, the user could ask, “Call the one on Rainbow Rd.,” “Call the bottom one,” or “Call this number (present onscreen).” Siri can’t perform this second part, but with ReALM, this language model could understand the context by analyzing on-device data and completing the query.
Ferret LLM
This paper explains how a multimodal large language model can understand user interfaces of mobile displays. The researchers say they have advanced in MLLM usage but still “fall short in their ability to comprehend and interact effectively with user interface (UI) screens.”
This assistive assistant is still far from being released. But once Apple masters it, it could be integrated alongside ReALM model.
BGR will update this guide as we learn more about Apple’s AI efforts.
