Apple Announces ARKit 3 With Immersive People Occlusion Feature, RealityKit, and More

Apple at WWDC 2019 today made a series of augmented reality announcements headlined by ARKit 3.


ARKit 3 features a new immersive "People Occlusion" feature that allows virtual objects to be placed in front and behind people in real time, as Mojang demonstrated on stage with a preview of its augmented reality game Minecraft Earth.

Apple's new RealityKit SDK provides developers with tools for photorealistic rendering, environment and camera effects, animation, physics, and more.

Stay tuned for updates…

Related Roundup: WWDC 2019

This article, "Apple Announces ARKit 3 With Immersive People Occlusion Feature, RealityKit, and More" first appeared on MacRumors.com

Discuss this article in our forums

ARKit 2.0 Will Let Two iPhones See the Same Virtual Object

iOS 12, set to be unveiled at the Worldwide Developers Conference on Monday, will include ARKit 2.0, an upgrade to the existing ARKit 1.5 SDK that's available to developers to allow them to build augmented reality experiences in their apps.

ARKit 2.0, according to a new report from Reuters, will include a feature that's designed to let two iPhone users share an augmented reality experience, with the same objects displayed on multiple screens.


This is in line with previous rumors from Bloomberg that have said Apple is working on both multiplayer augmented reality gameplay and object permanence, which would allow a virtual object to remain in place across multiple app sessions.

Apple is aiming to allow two people to share data so they can see the same virtual object in the same space via each individual device, with Apple designing the feature in a privacy-friendly way.

Apple's multiplayer system, unlike similar offerings from Google, does not require users to share scans of their homes and personal spaces, working via a phone-to-phone system.
Apple designed its two-player system to work phone-to-phone in part because of those privacy concerns, one of the people familiar with the matter said. The approach, which has not been previously reported, differs from Google's, which requires scans of a player's environment to be sent to, and stored in, the cloud.
Full details on how Apple's multiplayer augmented reality system will work are unknown, and it's not yet clear if it works with three or more players. Apple will share more information on the feature on Monday.

Augmented reality has been a major focus for Apple over the course of the last two years, with Apple CEO Tim Cook calling AR "big and profound." "We're high on AR in the long run," Cook said in 2016.

Apple unveiled its first augmented reality product, ARKit, with iOS 11 at WWDC, and has since made improvements to the feature with the launch of ARKit 1.5 in March as part of iOS 11.3. ARKit brought mapping for irregularly shaped surfaces, vertical surface placement, and object and image recognition. With the additional changes coming in iOS 12, developers should be able to do a whole lot more with augmented reality.

Related Roundup: iOS 12
Tag: ARKit

Discuss this article in our forums

Gene Munster Shares WWDC Predictions: Beats Product With Siri Integration, Improvements to AI and AR

Ahead of next week's Worldwide Developers Conference, Loup Ventures analyst Gene Munster today shared his predictions for the features and services that Apple will unveil during the event.

Munster expects Apple to debut new Siri, AR, AI, and Digital Health functionality, including a Beats-branded accessory (presumably a speaker) that includes Siri integration, much like the HomePod. Some of Munster's predictions have been previously covered in rumors shared by Bloomberg's Mark Gurman, but Siri integration in a lower-cost Beats product is a new prediction.

We expect Monday's keynote to be highlighted by extending the reach of Siri (most likely adding new domains, opening HomePod to more capabilities, and integrating Spotlight), along with additional AI tools (new Core ML extensions).

We also anticipate new features around digital health (privacy and device management) and ARKit (development tools).

Expect Siri integration with Beats.

Collectively, these announcements advance the ease of use and intelligence of Apple's mobile and desktop experiences.
According to Munster, Apple may be planning introduce a $250 Beats-branded product that will offer Siri integration similar to the HomePod, allowing Apple to "advance its digital assistant ambitions" with a more affordable option. Apple currently sells a Beats Pill+ speaker for $179.95, and the device has not been updated in some time.

Apple is going to announce a new "Decade Collection" at WWDC according to a Best Buy leak, but that collection is limited to existing headphones in new colorways and does not appear include new products or a new speaker. It's possible that Apple does have a new Beats product ready to unveil, and a recent somewhat sketchy rumor did suggest that Apple's "low-priced" HomePod would be under the Beats by Dre branding.

That rumor would make some sense if Apple is indeed planning on introducing a Beats-branded speaker product that includes Siri integration. Siri competitor Alexa is available as an option on many speakers outside of Amazon's own, and Apple could be planning to follow in Amazon's footsteps. Obviously, though, it's not clear if the rumor and/or Munster's prediction are accurate.

Munster has several other predictions for features and services coming at WWDC. Specifically, he expects Apple to introduce new Siri domains, with support for "things like navigation and email" and integrated Spotlight Search to better improve Siri's performance compared to Alexa and Cortana, the AI assistants used by Amazon and Microsoft, respectively.

Munster also believes Apple will introduce new domains for CoreML, the machine learning SDK that Apple introduced with the launch of iOS 11. Munster doesn't offer details on what the new domains might be, but at the current time, CoreML offers features for developers like real time image recognition, search ranking, text prediction, handwriting recognition, face detection, music tagging, text summarization, and more.

Previous rumors have said Apple plans to introduce support for multiplayer augmented reality games, and Munster believes Apple will also introduce "subtle new developer tools" to improve AR development and lead to more compelling AR apps.

Similarly, rumors have indicated Apple is working on a Digital Health tool that will let parents better monitor the amount of time children are spending on iOS devices. Munster says Apple could also include additional features that notify users when data is being shared with developers and new device management features aimed at curbing "screen time and digital anxiety."

Munster's full range of predictions for WWDC can be read on Loup Ventures, and our iOS 12 roundup contains all of the other iOS-related rumors that we've heard thus far.

Related Roundup: HomePod
Buyer's Guide: HomePod (Buy Now)

Discuss this article in our forums

‘Star Wars: Jedi Challenges’ iOS App Updates With ARKit Support for Holochess Mode

When Apple first announced its ARKit developer platform, one of the examples it gave was an image that showed someone playing Dejarik, a holographic board game from the Star Wars universe. The game has been available in an augmented reality experience on the iOS "Star Wars: Jedi Challenges" app [Direct Link], but it required pieces of hardware -- namely the Lenovo Mirage AR headset -- that currently costs $149.99.


This week Disney updated the Jedi Challenges app with ARKit support, removing the requirement of a Lenovo Mirage AR headset and letting anyone with an iPhone or iPad running iOS 11 play Dejarik Holochess for free (via Gizmodo UK). ARKit users are able to access the full Holochess game mode, with 18 levels taking place across six planets and including eight unlockable creatures with unique abilities. Lenovo says that Holochess mode includes around two to three hours of gameplay.
Star Wars: Jedi Challenges adds ARKit compatibility with this update. Users with an Apple device running iOS 11 can now experience the magic of augmented reality directly from their mobile device. Access the full Holochess game mode from Star Wars: Jedi Challenges including 18 levels across 6 planets and 8 unlockable creatures with unique special abilities.
Last month, Sensor Tower reported that iPhone and iPad owners worldwide have downloaded more than 13 million ARKit-only apps since the platform launched in September 2017. Games remain the dominant category for ARKit-only apps -- defined as "expressly using" Apple's framework, unlike Jedi Challenges -- having grown from representing 35 percent of downloads one month after iOS 11's launch, to 47 percent today.

Back in January, Apptopia claimed that developer use of ARKit had slowed down since it debuted in September, with growth steadily declining since the official launch. Apptopia's numbers suggested that 300 ARKit-related apps launched in September, around 200 came out in October, and 156 were released in November. The number climbed above 160 for December, but developers are said to still be figuring out the best use cases for augmented reality in general, with many ARKit-enabled apps designated as either games with AR modes attached onto them, entertainment and photo apps, or utilities.

Although ARKit has removed the need for a headset in Holochess mode, anyone who wants to partake in Star Wars: Jedi Challenge's Lightsaber Battles and Strategic Combat games will still need the Mirage headset. While Jedi Challenges is also available for Android smartphones, support for Google's ARCore has not yet been announced.


Discuss this article in our forums

ARKit-Only Apps Exceed 13M Global Downloads Since Launch as Games Remain Most Popular Category

Since ARKit debuted within iOS 11 on September 19 last year, iPhone and iPad owners worldwide have downloaded and installed more than 13 million ARKit-only apps. The data comes from Sensor Tower, which broke down the most popular categories of augmented reality apps fueled by ARKit, the top 10 free and paid apps, highest grossing apps, and more.


Games remain the dominant category for ARKit-only apps -- or those apps built "expressly using" Apple's framework -- having grown from representing 35 percent of downloads one month after iOS 11's launch, to 47 percent today. The second place category, Utilities, decreased from 19 percent last October to 15 percent in March. Rounding out the top six were Entertainment, Lifestyle, Photo & Video, and Education.

Charts via Sensor Tower

Sensor Tower pointed out that for the games-specific charts, the top spots of all three sections (free, paid, grossing) were "still occupied by many of the same titles that found success several months ago," suggesting not much variation in ARKit-only gaming on the App Store. On the other hand, when the researchers looked at non-game apps they saw multiple newcomers rising on the charts, like LEGO AR Studio -- "a testament to the popularity of kid-focused AR content on the App Store thus far."
From what we’ve seen in our latest analysis, ARKit-only apps only continue to grow in terms of installs and the number of experiences available to users. This isn’t including the numerous ARKit-compatible apps that have added some degree of AR functionality in the past six months and have pushed the number of AR apps on the App Store well beyond 2,000 to date.

There’s clearly substantial room for growth in terms of user base and revenue, but also ambition when it comes to this burgeoning category, and the apps above have built a solid foundation for what’s to come, especially as the capabilities of ARKit evolve and expand with future versions of iOS.
Popular free game "AR Dragon" retained the top spot as the most-downloaded free ARKit-only app and game during the framework's first six months of availability. Other augmented reality apps that remain popular on the App Store include IKEA Place (#2 free apps), AR MeasureKit (#5 free apps/#5 top grossing), and CamToPlan Pro (#1 paid apps). For both paid and grossing categories, ARKit app downloads are leaning "predominantly" to the Utilities category.


Apple CEO Tim Cook has described augmented reality as "profound" in the past, claiming that Apple is in a "unique position" to lead when it comes to the technology. He thinks that AR will become "as key as having a website" for brands, and sees ARKit as the start of something much bigger: "This is very much like in 2008 when we fired the gun in the App Store. That's what it feels like to me and I think it will just get bigger from here."


Discuss this article in our forums

Apple to Offer Presentation on ARKit at This Year’s Game Developers Conference

Apple will be hosting a session at this year's Game Developers Conference for the first time, offering an introduction to ARKit, its augmented reality platform for developers.

The session will be presented by Michael Kuhn, who leads Apple's ARKit engineering team.


Entitled "Introduction to Apple's ARKit: Best practices and recent updates," the talk will cover core concepts of the ARKit framework and the ARKit API. It's designed to teach game developers how to get started with ARKit, and it will cover ARKit best practices.
This session introduces core concepts of the ARKit framework, it's underlying principles, and the ARKit API. It explains how to get started with ARKit using the different tracking and scene understanding capabilities as well integration into rendering/game engines. The session also highlights best practices for AR like starting an experience, placing objects in the real world, interacting with them and implications for games. In addition it explains basic concepts and challenges of AR and Computer Vision to help avoid common pitfalls and allow the creation of great experiences.
Apple has not previously offered developer sessions at GDC, but this is the first GDC since the launch of ARKit and Apple is likely hoping to get more game developers interested in implementing augmented reality features.

ARKit was introduced as part of iOS 11 back in September of 2017, and since then, developers have incorporated augmented reality features into more than 2,000 apps. Major improvements are coming to ARKit with the launch of iOS 11.3 and ARKit 1.5, which may come out right around when GDC takes place and will likely be a topic of discussion.


ARKit 1.5 can map irregularly shaped surfaces for better detection of ambient surroundings, it can recognize and map vertical surfaces like walls and doors, and it includes an image detection feature that works on everything from movie posters to bar codes.

The 2018 Game Developers Conference will kick off on March 19 at the Moscone Center in San Francisco, and it will last until March 23.

Tag: ARKit

Discuss this article in our forums

Developers Demo Augmented Reality Improvements Coming in iOS 11.3

The iOS 11.3 update, seeded to developers and public beta testers last week, introduces ARKit 1.5, an upgraded version of the set of tools developers can use to create augmented reality apps for the iPhone and the iPad.

As it turns out, ARKit 1.5 can do a lot of neat things. It can map irregularly shaped surfaces for better detection of your surroundings, and it can also recognize and map vertical surfaces like walls and doors, so you can use AR to place and detect items on walls.

Over the course of the last week, developers have been testing out ARKit 1.5 and sharing short demo videos on Twitter, providing a look at just what will be possible with augmented reality apps when iOS 11.3 is available.


Vertical surface detection, for example, is shown off in the video below. A realistic-looking tunnel is projected on a wall, and while this doesn't have any immediate usage implications, it's a useful demo of how ARKit sees walls in iOS 11.3.


An example of how vertical plane detection can be used in augmented reality games is demonstrated in the video below, where creatures projected into open space take advantage of the area around them.


Another demo adds virtual artwork to a blank wall, a concept that could potentially be used in an art gallery or museum where art is invisible without a smartphone.


Vertical plane detection is used in the video below to show a realistic-looking virtual cockatoo coming through a window and landing on a windowsill.


In addition to mapping oddly shaped spaces and recognizing vertical surfaces, ARKit 1.5 also includes image detection features that work on everything from movie posters to barcodes, as demoed below. In the future, you might be able to scan a barcode with ARKit to get a virtual popup of nutritional information, calories, and more.


Image detection could be useful in settings like art galleries and museums, where visitors could use it to scan paintings and exhibits to receive more information, as shown off in the video below.


Though not visible in the demo videos shared by developers, ARKit 1.5 also introduces a higher camera resolution, so passthrough video is 1080p rather than 720p, and there's also support for autofocus capabilities, another feature that will improve the augmented reality experience on iOS devices.

Recent data has suggested that the ARKit framework has seen only modest adoption from developers and stagnating growth since its debut in iOS 11, but improvements like ARKit 1.5 may change that in the future. Augmented reality on iOS devices is still in its infancy and it will take time for developers and users to discover the best real-world use cases for the technology.

Apps using ARKit 1.5 will be available starting this spring when iOS 11.3 is released to the public.


Related Roundup: iOS 11
Tag: ARKit

Discuss this article in our forums

iOS 11.3 Coming This Spring With New Animoji, Vertical ARKit, Health Records, Battery Info, and More

Apple today previewed iOS 11.3, its next major iPhone, iPad, and iPod touch software update. The first beta will be seeded to developers later today, followed by a public beta soon, ahead of an official release this spring.

iOS 11.3 introduces new Animoji on the iPhone X, including a lion, bear, dragon, and skull. There will now be 16 characters to choose from in total, including existing ones like a pig, fox, chicken, pile of poo, and robot.


iOS 11.3 will feature ARKit 1.5. In addition to horizontal surfaces like tables and chairs, Apple's updated augmented reality platform will now be able to recognize and place virtual objects on vertical surfaces like walls and doors, and more accurately map irregularly shaped surfaces like circular tables.


ARKit 1.5 can find and recognize the position of 2D images such as signs, posters, and artwork, and integrate these real-world images into augmented reality experiences, such as bringing a movie poster to life. In addition, the view of the "real world" will now be in 1080p HD, up from 720p currently.

The software update will introduce Business Chat, a new way for users to communicate directly with businesses within the Messages app. This feature will launch in beta following the public release of iOS 11.3 this spring, with support from select businesses, including Discover, Hilton, Lowe's, and Wells Fargo.

With Business Chat, it's easy to have a conversation with a service representative, schedule an appointment or make purchases using Apple Pay in the Messages app. Business Chat doesn’t share the user’s contact information with businesses and gives users the ability to stop chatting at any time.
In the Health app on iOS 11.3, users will be able to view health records, including available medical data from multiple providers like Johns Hopkins and Cedars-Sinai. The data is encrypted and protected with a passcode.


iOS 11.3 will provide users with an iPhone 6 or newer with more information about the health of their device's battery, including a recommendation if it needs to be serviced. In the same menu, it will also be possible to see if Apple's power management feature is active and turn it off if desired.

Apple says the battery and power management features will be coming in a later iOS 11.3 beta release, so they won't be available today.

Other new iOS 11.3 features include more prominent placement of music videos in Apple Music, a new Video section in the "For You" tab of Apple News, and support for Advanced Mobile Location (AML) to automatically send a user's current location when making a call to emergency services where supported.

The first beta of iOS 11.3 will be seeded to developers later today, followed by a public beta soon. The software update will be released to the public this spring for iPhone 5s and newer, all iPad Air and iPad Pro models, the fifth-generation iPad, iPad mini 2 and newer, and the sixth-generation iPod touch.

Related Roundup: iOS 11

Discuss this article in our forums

Apple Reportedly Working on 3D Sensor System for Rear Camera in 2019 iPhones

Apple is developing 3D depth sensing technology for the rear-facing cameras in its 2019 iPhones, according to a new report by Bloomberg on Tuesday. The 3D sensor system will be different to the one found in the iPhone X's front-facing camera, and is said to be the next big step in turning the smartphone into a leading augmented reality device.

Apple is evaluating a different technology from the one it currently uses in the TrueDepth sensor system on the front of the iPhone X, the people said. The existing system relies on a structured-light technique that projects a pattern of 30,000 laser dots onto a user's face and measures the distortion to generate an accurate 3D image for authentication. The planned rear-facing sensor would instead use a time-of-flight approach that calculates the time it takes for a laser to bounce off surrounding objects to create a three-dimensional picture of the environment.
The existing TrueDepth camera would continue to be used in the front-facing camera of future iPhones in order to power Face ID, while the new system would bring the more advanced "time-of-flight" 3D sensing capability to the rear camera, according to the sources cited. Discussions with manufacturers are reportedly already underway, and include Infineon, Sony, STMicroelectronics, and Panasonic. Testing is said to be still in the early stages, and could end up not being used in the phones at all.

With the release of iOS 11, Apple introduced the ARKit software framework that allows iPhone developers to build augmented reality experiences into their apps. The addition of a rear-facing 3D sensor could theoretically increase the ability for virtual objects to interact with environments and enhance the illusion of solidity.

Apple was reportedly beset with production problems when making the sensor in the iPhone X's front-facing camera, because the components used in the sensor array have to be assembled with a very high degree of accuracy. According to Bloomberg, while the time-of-flight technology uses a more advanced image sensor than the existing one in the iPhone X, it does not require the same level of precision during assembly. That fact alone could make a rear-facing 3D sensor easier to produce at high volume.

Late last month, oft-reliable KGI Securities analyst Ming-Chi Kuo claimed that Apple is unlikely to expand its front-facing 3D sensing system to the rear-facing camera module on iPhones released in 2018. Kuo said the iPhone X's 3D sensing capabilities are already at least one year ahead of Android smartphones, therefore he believes Apple's focus with next year's iPhone models will be ensuring an on-time launch with adequate supply.

Related Roundup: iPhone X
Buyer's Guide: iPhone X (Buy Now)

Discuss this article in our forums

Amazon Updates iOS App With ARKit Support for Augmented Reality Holiday Shopping

Amazon is preparing its customers for the busy holiday shopping season, posting a detailed plan today on how it aims to help you get the items you want through Alexa voice shopping, curated gift guides, and a new "AR View" in the official Amazon iOS app. AR View was built using Apple's augmented reality developer framework called ARKit, and the company said it's meant to help customers "make better shopping decisions."


Similar to IKEA Place, Amazon's AR View lets you place virtual versions of real-world products sold on Amazon right into your home, helping you decide whether or not you like the item in a specific living space before you buy it. AR View can be found on the camera icon in the Amazon iOS app, which now has a new "AR View" option. Then you can select from "thousands of items" sold on Amazon, which includes home furniture, toys, Echo products, kitchen electronics, decor, and more.
Amazon’s latest augmented reality offering within the Amazon App launched today for customers with iOS 11 installed on their iPhone 6S or later. Using Apple’s ARKit, AR view helps customers make better shopping decisions by allowing them to visualize the aesthetic and fit of products in their own living space. Customers simply open the Amazon App, click on the camera icon and choose AR view.

They can then select from thousands of items – from living room, bedroom, kitchen and home office products to electronics, toys and games, home décor and more. Whether customers are buying a sofa or a kitchen appliance, they can overlay it onto their existing living space, move it and rotate it to get a full 360-degree peek in a live camera view to make sure it fits their style and aesthetic.
ARKit debuted in iOS 11 in September, and you'll need an iPhone 6s or later running iOS 11 to use the new AR View feature in Amazon's iOS app. Target also launched an augmented reality shopping feature last week, but it wasn't in its iOS app and instead debuted in the Target mobile website. Additionally, Target's new "See It In Your Space" option doesn't use ARKit.


Amazon is available to download for free from the iOS App Store. [Direct Link]

Tags: Amazon, ARKit

Discuss this article in our forums