A look at the future of Augmented Reality from #ARinAction

PPR visited the #ARinAction Industry Summit, which took place at the MIT Media Lab on January 16-17. This is a great event to not only see the future of AR, but also to learn how augmented reality is being applied in industry today. Here are some highlights:

  • Near-instant 3D model creation of interior spaces just a few years away? “In the next year or two there will be devices that can capture a 3D model of a room in a few seconds,” said Mark Billinghurst, Professor of Human Computer Interaction at the University of South Australia. If he’s right, and the models are accurate, this greatly reduces the requirements for 3D model creation in closed spaces, which could be a boon for AR as well as industrial IoT applications that require spatial data.

  • PTC’s Mike Campbell trotted out some industrial AR demos which were slick … yet didn’t look that convincing. For instance, one showed Ford engineers or designers wearing Hololens headsets and looking at data layered on top of a model sports car. It was neat, but it didn’t seem like the value delivered from this experience was superior to screen-based or paper alternatives.

  • PTC wasn’t the only one showing off unconvincing AR demos. One of the academic presenters had an AR tool for demonstrating math and physics concepts to students, which looked cool, yet also seemed complicated and costly. There are not many school systems that could realistically invest in hardware, training, and content to make this work for their students.

  • On the other hand, the DHL augmented reality system demonstrated by PTC seemed to be an effective solution for a real industrial use case. It is used to find and track items in a large logistics operation, and seems more promising, as it’s hands free and speeds the completion of specific tasks. This is similar to the Google Glass system used by AGCO to track completion of manufacturing tasks, and may even be competitive with other IoT logistics systems entering the market, such as ProGlove.

  • Speaking of Google Glass, it was barely mentioned at ARinAction. This surprised me … isn’t Google trying to reposition Glass for industrial use? But then Steven Feiner of Columbia University shared a piece of information that might explain why Glass was MIA from the conference: Google Glass is not really augmented reality: “Google Glass isn’t a true AR display,” Feiner said. “It can’t handle overlays, for one … Doesn’t have stereoscopic view, either.”

  • Solos AR glasses for cycling based on Kopin componentsInnovations from the military are making their way into industrial and consumer devices. John Fan, the CEO of military supplier Kopin presented an example – the heads-up displays used by F-35 pilots have led to technologies that can be used in AR-equipped firefighting helmets from Scott Sight. There was also a prototype cycling AR display (see inset photo) that used Kopin components. Fan shared a relevant observation about helmet-based AR: “Basic premise: humans don’t want to wear things on their heads,” he said, explaining that the technology has to deliver real value to get them to wear headsets … and keep them on. This is true for military and public safety uses in which lives are stake, but perhaps less so for other applications.

  • “Interim devices” are the trend in augmented reality for the next 10 years, according to futurist and author Charlie Fink. “For AR to realize its potential, it needs to know you, and where you are, and it has to have access to data,” Fink said. “We’re not there yet.” He stated that a lack of infrastructure and key breakthroughs are holding back AR.

  • There is a lot of froth in the marketplace. Analyst Tim Merel, a former engineer, noted the arrival of ARKit and other AR tools from Facebook, Tencent, and others, which fuels interest in the field. Nevertheless, “there are even more VCs than there are startups,” he said. Merel noted “mobile AR still at the very early stages” and exits will be relatively small in the near term, as dominant companies have yet to emerge.

Endpoint: It was interesting to see some of the trends and examples in augmented reality, but at the same time there seems to be a lot of wishful thinking among some of the technologists, academics, and investors, not to mention a fair number of research projects or proof-of-concept applications that won’t go anywhere. Just because a technology is cutting edge and dramatic doesn’t mean it will be useful out in the field … or that humans will want to use it.

Interview: ProGlove’s Jonas Girardet

ProGlovePPR recently interviewed Jonas Girardet, the COO and cofounder of Munich-based ProGlove, which makes a wearable IoT device for manufacturing, logistics, and other specialized industrial uses. The system is already in use at BMW, Lufthansa Technik, and other manufacturers and retailers, and recently exhibited at CES. The following interview was edited for clarity and was first published in the PPR newsletter earlier this month. 

PPR: What does the ProGlove do?

Girardet: ProGlove connects the worker to the industrial IoT. We develop smart gloves that enable manufacturing and logistic staff to work faster, safer and easier. If you go to a factory, you’ll see more barcode scanners than any other digital device, such as tablets, smartphones, notebooks, or PCs. Our smart gloves have a connected barcode scanner comfortably attached to the back of the glove to work and scan at the same time and to help workers become more efficient. It makes fewer mistakes when scanning, compared to a normal handheld barcode scanner.

Integration is quite easy, as it’s just text that’s transmitted. It’s plug and play — basically, there is no IT work involved which is nice because the sales cycle gets shorter. And of course it’s nice for the customer, because he can basically rip off the existing scanner, put on the ProGlove, and continue working. You can do that during a coffee break. You don’t have to stop the line, or do a 12-month ERP project.

PPR: Is it complementary to other wearable IoT systems, such as Google Glass?

Girardet: We created ProGlove because of the fast integration and the immediate value to the customer. Today you need to grab the scanning gun, do the scan, put away the scanner, and then do your work. With ProGlove, it’s basically part of your normal movement, so with every scan, you roughly save four seconds. But you have thousands of scans per day. And that’s why production process managers really appreciate the idea, and understand the benefit. And of course we see it as complementary to Google Glass.

PPR: How is the platform being expanded?

Girardet: We will have more hardware products, and we are now developing software products on this hardware platform, that will connect the human worker with industrial IoT.

For 150 years, manufacturing and logistics has been optimized for efficiency. Now we have these highly automated lines and robots. But a robot is not made for a product lifecycle of three months, or a lot-size change within two days from one million to ten million. But you can actually do that with human workers. The downsize nowadays is that human-related work is kind of a black box. The problem is, there is no communicated status of the current work and progress, because the worker are not connected to the rest of the factory. They are left on their own with their fixed tasks. We can improve flexibility and reliability if we change this. not When we connect the human to the IT systems in the factory, we can give the worker real time guidance of his next tasks to do.

PPR: How does connectivity work?

So at the moment we are using a proprietary standard. The most obvious choice would be Bluetooth, but what we learned is that Bluetooth is not widely accepted in industry because it’s operating in the 2.4 Ghz band. Big OEMs try to avoid bringing any device into the industry that is operating in that band. ProGlove has an access point in a little box that connects with your PC or to your network, and then vis Sub 1 Ghz to Mark, which is the scan module.

It’s operating at 915 Mhz  in the U.S. and  868 megahertz in Europe, but the standard itself is a proprietary standard.

PPR: Are you planning to use other LPWAN technologies?

Girardet: We are definitely looking into Bluetooth, because Bluetooth has evolved. When I talked about the spectrum problems, that was Bluetooth 2.1. Now the technology has evolved and is more stable. And we have customers, especially in the fast-moving consumer goods category, and they don’t they don’t have that problem of too many device conflicts because they have only scanners — there is no machinery or tools operating in these frequencies. And of course for us it’s about software that will be able to integrate deeper into manufacturing systems and ERP systems. From a pure hardware standpoint, it will always be Bluetooth or WiFi.

PPR: So ProGlove basically is replacing a standard barcode scanner. But in the future, the platform will bring the data or maybe some other information that’s being captured into other types of systems.

Girardet: That’s the vision. The first step is hardware, and barcode scanning. The second step will be connectivity, so you connect the worker with the system. The worker with the warehouse management system, and the warehouse management system with the worker. So you scan the barcode, and then immediately determine if it’s the right barcode or not. And then you can start to think about data flow.

Our customers at the moment are just thinking about step one. They are starting to think about step two. The data is definitely part of the vision – the human hand can generate a lot of data.

PPR: Talk a little bit about the number of installations or the verticals that you’re really heavily present in.

Girardet: We are from Munich, Germany, and the automotive industry is really strong there. We started to develop our product with BMW and Festo. You can say, almost every BMW from a German factory is built with ProGlove.

So the majority of our customers are in the automotive industry or their suppliers, but also fast-moving consumer goods, such as Rewe-Penny (German Wal-Mart) in Germany, which are using our products in supermarket warehouses. When a supplier ships yogurt to the warehouse, they want to be sure it’s the right kind of yogurt, and the right amount of yogurt gets shipped.

There is also pharmaceuticals. All over the world, laws have changed, so they really need to document more information, such as where the pharmaceutical goods come from. That’s why they scan a lot of barcodes.

As for the number of installations, it’s a few hundred warehouses and manufacturing sites in Europe that are using our product. Germany is our core market, but also the UK, Eastern Europe, and France. But now we are getting ready to jump into the U.S., one of the biggest manufacturing markets. The product is already used here, at the pilot stage.

ProGlove BMW
ProGlove in use at a BMW parts warehouse.

PPR: Does the ProGlove require a systems integrator?

Girardet: Basically it’s plug and play. You can plug it in next to a preexisting barcode system. And it can work with big industrial OEMs.

You can order the US version, we will ship it to you, and you can install it. However, normally we work together with our champions who want to work with the product, so we will do a proof of concept with them. The champion might buy a few units for two or three stations, which will generate a lot of data to see how much time is being saved, economic improvements, and worker acceptance. He can then convince IT, QA, and finance. And then it can scale.

PPR: Can you talk about efficiency metrics?

Girardet: We have the numbers confirmed by OEMs in Europe that it’s basically bringing down the duration of barcode scanning by 40 percent.

Of course, barcodes are only part of the process. But if you think about an auto manufacturing site that has a cycle time of 50 seconds, every 50 seconds the car moves, and then the operator does the same step again. And when they are installing an air bag, they basically have three or four things to do and one these things is scanning a barcode. There might be two barcodes, one on the car and one on the part. And one barcode scan takes eight seconds. And you basically save 40 percent of these 16 seconds, out of a 50-second cycle time. That’s really a lot of time.

That’s really why we have big returns on investments for our customers. There’s always a case for ‘Are you saving money? Are you really more efficient?’ You don’t buy it because it is cool, fancy stuff. You buy it because it is more efficient.

PPR: Last question: where do you see this technology in five years?

Girardet: In manufacturing, and supply chains, there is so much value being added compared to the consumer sector. That’s why I think wearables will really play a massive role in the future of the industrial world. Then of course one of these things will be Glass, something that’s in your eyesight, and the second thing will be in the form of a glove, or on your hand. Think about displaying more information, and also having feedback on the hands.


CES 2018 trends: AI, VR/AR, and a hint of 5G

The annual Consumer Electronics Show in Las Vegas isn’t just a gathering of gadget and toy manufacturers. CES 2018 had a lot of presentations, talks, and displays by vendors targeting industrial users. Here are some other CES trends worth noting:

AI assistants: Google dominated CES with announcements related to Home and other AI-powered assistants. While Google’s voice-controlled assistants don’t seem very relevant to industry, take the long view: Just as the Web and smartphones burrowed into industry after conquering the home, so will voice-controlled AI. Google, Amazon, and others are pouring billions into AI R&D, and the end result will be much more than ordering pizzas from your couch. Imagine voice-controlled devices, status updates, or diagnostics on the factory floor or out in the field — or synthetic voices that are indistinguishable from real humans. The physical form factor will surely be different, but it will be a game changer for many industrial users.

VR/AR: 3D graphics have already had an impact on certain areas of industry (CAD and some emerging uses of Google Glass spring to mind) but I am skeptical that the latest generation of VR headsets and haptics technology making much headway into industry. I’ve been around long enough to see misguided 3D hype lead corporate customers down the wrong path (anyone remember Second Life?) and I suspect the latest crop of VR technologies will remain more of a consumer phenomenon. Industrial operators aren’t going to put on a full-immersion VR headset, although there may be some applications for remote operation, evaluation, and training (as healthcare startup SimForHealth demonstrated at CES). AR looks more interesting, and Glass shows that there are some niche applications that can help companies save money and time.  

Smart Cities. There were reportedly more “smart city” vendors than companies selling gaming products or drones. The displays around smart cities had lots of eye candy when it came to autonomous vehicles and IoT-enabled homes, but a less-visible technology was the recently approved 5G standard. Qualcomm, Samsung, Ericsson, and other companies were hyping the heck out of 5G, but on the show floor there wasn’t much hardware to show in the smart city pavillion or elsewhere. That will likely change next year, as vendors bring their 5G-capable devices to the show, not only for the consumer market but also industrial uses.

Joe Weiss on industrial security standards: “This is about facilities being destroyed and people being killed”

PPR interviewed Joe Weiss, managing partner of Applied Control Solutions and the author of Protecting Industrial Control Systems from Electronic Threats, on a new initiative by the ISA99 committee of the International Society of Automation to create security standards for industrial sensors, actuators, and drives. In the ISA-95 reference architecture, these devices and their controllers correspond to level 0 and level 1. The interview has been edited for clarity.   

PPR: What’s the implication of billions of IoT devices coming online in industrial settings?

Weiss: I have some problems with the estimates. They’re mixing fitbits and refrigerators with industrial process sensors and controllers. A process sensor in a power plant or a pipeline is very different than a home sensor. The vast majority of new sensors are not for industrial applications, and that’s getting lost.

There’s certainly a big number, but it’s not the billions that many IoT think tanks keep mentioning.

PPR: Is Mirai a sign of what’s to come in the future, or do you think that companies and governments have a good handle on IoT security?

Weiss: They don’t. ISA99 is starting a new working group on what’s called Level 0-Level 1, because none of our major industrial vendors makes secure, authenticated process sensors, actuators, or drives.

PPR: But the standards are in place, aren’t they?

Weiss: No! That’s exactly why we’re starting the ISA99 Working Group. What’s happened is very simple: IT has led cyber security. So cyber security has been all about the networks, as that’s what IT knows. The process sensors actuators and drives are engineering systems. Consequently, they have not been given security consideration.

We’re talking pipelines, power plants including nuclear, trains, manufacturing, water systems, and buildings. Security hasn’t reached what you measure and what you control. In other words, the things that actually cause control to happen, such as a motor or a valve or a damper, doesn’t yet have security or authentication.

PPR: When you hear vendors talking about their security, actually what they’re talking about is IT security, then.

Weiss: What they’re talking about is the network. And the things you keep hearing about are IT/OT convergence. But that doesn’t include the sensors, actuators, or the drives (Level 0,1). Yet, it’s the level 0,1 devices that are most critical for safety.

PPR: Why has it taken so long for ISA or other organizations to turn to the specific problems you’re talking about?

Weiss: Because Level 0,1 is engineering not networking. And the engineering/IT divergence goes back to 9/11 as I mentioned in my December 11th blog on a brief history of ICS cyber security.

The turning point was 9/11, because before 9/11, the organization that owned these systems also “owned” cyber security of those systems. However, on 9/12, the day after 9/11, cyber security became national security and cyber security moved to IT. As IT generally does not understand ICS, we’ve been suffering with their view of cyber security since.

My analogy of, ‘if you’re a doctor and you can’t trust your temperature or your blood pressure readings, how can you make a diagnosis?’ describes the culture problem that continues to exist. All of IT cybersecurity is about making the diagnosis. Almost nobody is asking, can we even trust the sensors which is at the heart of IoT. Yet IoT is all about more sensors using high-powered data analytics. If you can’t trust your sensors, what does that mean about IoT?

PPR: Who will be participating in the Level 0-Level 1 initiative?

Weiss: As of this morning’s (December 14th) call,  we had vendors, end –users, consultants, and even a government representative – from Germany. But basically Level 0,1 devices are not in the “sweet spot” for most cybersecurity firms. What cyber security organizations have done up till now is assume that whatever the sensors are telling you must be correct. And what the cyber security organizations have done is spend their efforts on the networks.

PPR: Could you name some of the vendors that are participating?

Weiss: Today was the initial call and we had representatives from Siemens, Schneider, Honeywell, GE, and Cisco participating. I would hope all of the ICS vendors and cyber security practitioners will eventually participate as the lack of Level 0,1 security is a risk issue for everyone in this space.

And let me give you a scary story. I gave a presentation at the this year’s DEFCON conference on the lack of level 0,1 security. On October 24th as I was getting ready to give the state of the state presentation at the Security Week ICS Cyber Security Conference, I happen to look at my LinkedIn account.  I had a ‘like’ from that Defcon presentation and it turned out to be from a senior engineer in Iran – they know! Think about that as today Fireeye identified a safety system being hacked in Saudi Arabia and I think it is safe assumption to say it was from Iran.

The lack of Level 0,1 security hits at the heart of IIoT. It also hits at the heart of the industrial cloud because every cloud vendor assumes that all sensor input is secure and authenticated.

PPR: So let’s assume that ISA 99 kickoff meeting goes well, and there’s an initiative to establish standards for these different types of things.

Weiss: The first task is to review the existing standards and identify gaps with respect to Level 0,1 devices. Additionally, Level 0,1 device issues are not just cyber security but also affect process safety. Level 0,1 devices are what the network cyber security doesn’t adequately address. Yet, this is where facility and personal safety are at risk.

PPR: Now as you know an initiative like this is going to take many years to actually start getting somewhere.

Weiss: You’ve got to start somewhere. I had identified the Level 0,1 issue as far back as 2000 when I helped start the cyber security program for the electric utilities. And it’s kind of a damning statement to say it’s now almost 2018 and we haven’t even started.

The lack of Level 0,1 cyber security is a major risk which has nothing to do with somebody stealing data. This is about facilities being destroyed and people being killed. So I would expect the vendors are going to actively involved because they are at risk too. I believe they want to know what’s the right thing to do. This is not a me vs. them situation. This is, ‘whoops, we forgot something.’

PPR: In the meantime, what does a manager of a power plant or a factory do?

Weiss: We’ve got to educate people to even look under the rock. What they do now, is they pick up their feet and step over the rock. This means understanding the unique issues with ICS cyber security starting at Level 0,1. We also need technologies to monitor Level 0,1 devices BEFORE they become Ethernet packets. So far, there is at least one company working in this area.

What they assumed was it’s the Internet that’s the problem. The Internet is a problem with the network. Therefore, that’s where people went.

PPR: Many industries consider themselves to be vertically oriented.

Weiss: No. The same Rockwell programmable logic controller used in power plants are also used in water, oil, gas, chemicals, manufacturing, railroads, amusement parks, breweries, ships, and buses. Consequently, the approach ISA 99 took is develop cyber security standards and recommended practices that would be applicable for pipelines, etc. There is no difference! Consequently, the concern with the revelation today about the cyber attack of the Triconex PLC because this device is used in many industries including nuclear plants, refineries, chemical plants, water systems, etc.

PPR: Is the Government going to have anyone there?

Weiss: My co-chair is from the Oak Ridge National Laboratory. Other than my co-chair, the only government representative on today’s call was from Germany.

Weiss writes about industrial security at the Unfettered Blog.

Four industrial IoT & AI Predictions for 2018

PPR has some predictions for IoT and AI in the coming year. This article first appeared in the PPR newsletter – subscribe today!

#1: The year of turnkey IIoT solutions

Vendors have figured out that many industrial customers don’t have the expertise to plan and execute an industrial IoT rollout. In 2018, more and more vendors will partner to offer turnkey IIoT solutions combining hardware, software, systems, and services. Look for announcements from industrial vendors working with big names in IT counterparts and/or integrators throughout the year.

#2: IoT pilots get real

When it comes to pilot projects involving industrial IoT, 2018 is the year many firms will move beyond talking the talk and start walking the walk. The availability of turnkey solutions, not to mention heightened interest in IoT and related industrial automation technologies, will translate to more industrial IoT pilot projects in 2018. The pilots are more likely to take place at midsized and large firms with healthy balance sheets and mandates to innovate. These projects will not only help industrial companies evaluate the technologies, but will also create technological foundations and centers of internal expertise that can help broaden these pilots into full-fledged production systems in the years to come.

#3: Amazon ascendant

Industrial vendors and high-tech players in industrial IT including Rockwell, GE, and Microsoft will be dealing with a new player in town: Amazon. Not content with merely offering highly profitable cloud services to companies, Amazon wants to control various platforms that feed into its AWS cloud offerings. The FreeRTOS announcement in November was a sign of things to come (see Parsing Amazon’s FreeRTOS announcement: Interesting, “but kind of strange”). In 2018, look for Amazon partnering with key vendors, making new acquisitions, taking a more active role in standards-setting bodies, and announcing new products and services to bolster the attractiveness of the AWS ecosystem to industrial customers.

#4: Industrial ML apps: still a work in progress

Late last year PPR published “It’s time for a Machine Language reality check.” In 2018, we will hear more of the same hand-waving hype about industrial applications of ML, but the really important developments will be in the foundational technologies that will allow vendors to build industrial ML applications and services. Google is the big player in this space, but smaller companies with strong AI R&D capabilities may have some interesting announcements as well.

This article first appeared in the PPR newsletter on January 2, 2018.