Juniper Networks Marvis Update - Mini's Digital Experience Twins

0:09 uh very first right up front for us is is truly Marvis our Crown Jewel uh you

0:14 know this is uh I'm going to introduce Bob Friday Bob's the chief AI officer for uh Juniper Networks uh was obviously

0:21 the co-founder for M uh where stoked will start with Marvis and we'll be uh we'll be hidden off so Bob come on up

0:28 thank you Sadir you know as and Su say this is my favorite event of the year because this is where we talk about the

0:33 latest and greatest Innovations around Cloud aops networking now when Su and I

0:40 started this adventure you know it was really with the observation that there was a fundamental Paradigm Shift we still have

0:47 to keep all these boxes greens AP switches and routers got to keep all that stuff running but what we learned

0:53 from the customers was Wi-Fi was going from a nice tab to a must business critical and they wanted to make sure

1:00 whatever critical app they put on network whether it was going to be a consumer app or robot that that was going to have a great experience and

1:06 that was the cloud AI Ops Paradigm Shift yes we keep all these boxes green but we

1:11 got to really start focusing on why that user is having a bad experience regardless of it's one of our

1:17 boxes now someone talked about the AP the reason that we started the company with the access point and we built that

1:24 access point because we wanted to make sure that we're going to get the data we needed to answer the question of why

1:29 you're having a bad mobile experience bad Wi-Fi experience additionally we

1:35 remember Miss we also believe that indoor location and connectivity were converging and so we put a very special

1:41 B antenna to make sure that we had location and connected conversion since we started at Juniper you've seen us

1:48 basically extend Marvis across the access point the switch and the SD Wan

1:54 router and as we ingest more data this lets us get to more granular root causes

2:00 right that access point has a lot of information about the user state that sdw rer has a lot of information about

2:07 layer three and the application's running on it and last year what we brought to the party was Zoom this is

2:14 bringing the end in Loop to the end of the party right now we have complete visibility on what that actual end user

2:21 experience is going on for our zoom and we're going to be now talking about the innovations that we've done around our

2:28 continuous learning aspect what we're announcing new this year is starting to extend Marvis across these

2:34 Wan routers which is going to bring bring more Wan features in Marvis you know when you look at what's going on in

2:40 your network you know it's not usually the wireless problem the other big a here is the WAN connectivity what is

2:46 that mobile operator doing when something goes wrong half the time is somewhere in that Wan router in the

2:52 mobile operator Network that's connecting you to the internet the other big announcement that

2:57 we're bringing to the party this year is extending Marvin over the data center this is really bringing branch and data

3:04 center Better Together story these are for business critical apps like point of sale right you have a business critical

3:11 app running in your branch it goes down you need to immediately identify what layer of the network is causing a

3:18 problem is it in my Branch campus or is in my data center so this is the next critical Innovation that you'll see

3:24 coming down this year

3:30 and finally we're bringing Knack you know I think a lot of you have already seen that this was announced last year

3:36 this is our new Cloud Knack radius authentication Service and this is

3:41 really going to bring into Marvis more visibility on that preconnect you know when you use connecting when you have

3:48 the Miss Knack we're going to get better visibility on how that user experience is going so the take away from the site

3:54 is Paradigm Shift client to Cloud this is where the networking I think everyone has now realized it 10 years ago wasn't

4:00 quite so obvious now Clyde aops is the Talk of the

4:07 Town now I think heard the journey you know you've seen this slide I think I've

4:13 shown the slide for six seven years ever since we've been doing MFD and the reason this slide has not changed is

4:20 because the foundational pillars of cloud iops has not changed right and the

4:26 first element of this perod we talked about is data right and the reason we built that access point you know I think

4:32 the SS are everyone knows me I make a barrel of wine great wine starts with great grapes great AI starts with great

4:39 data and this year what we're bringing to the party is Marvis minis this is

4:45 probably the most exciting announcement of the year what this is bringing is what do you do when there is no users on

4:52 your network Marvis Min is basically the digital twin for your preconnect and

4:58 postcon user experience and we'll be talking more about this but this is what allows Marvis now to make sure that your

5:04 network is ready for when the business opens at 8:00 in the morning so this is

5:09 probably one of the critical announcements AI starts a data Marvis minis is bringing a whole new set of data when we do not have users on the

5:16 network to provide that data next big thing is really around this continuous learning experiment

5:23 right this is basically that s SLE service level experience which was

5:29 another fundamental shift when you look at the SLE inside of Marvis they're based on the user

5:35 minute you know 20 years ago when I did airspace it was all about keeping track of the controllers and APS and the

5:41 network elements we've moved that Paradigm now the fundamental unit is user minute you're keeping track of that

5:48 user State at all times every minute is that user having a good or bad experience on the network continuous

5:54 learning is starting to bring that experience to the to our party now and

6:00 we'll be talking more about the innovation in that area you know data science you know

6:06 we're all using the same data science algorithms in the industry we all have access to the latest and greatest data

6:12 science algorithms the big change here is really around bringing those out to

6:17 the SLE what we're doing with our customer support and our data science team right this is where Miss has really

6:24 differentiated itself organizationally bringing these two teams together and you'll see it in what we call our new

6:30 application SLE so this year you'll see a new application SLE page that brings continuous learning zoom and teams up to

6:38 the Forefront uh for our it users and then finally conversational

6:44 interface you know since we started this we've had conversational interfaces since 2018 you know we fundamentally

6:52 believe that natural Lang is the next Generation user interface in the industry in networking and other

6:58 Industries we'll be talking about the three critical innovations that we're bringing into Marvis right now to make

7:05 Marvis more humanlike you know how do we make it the interface of choice for interacting with your network and

7:11 getting information from your network and finally what you'll hear from West

7:16 and sun is what are the latest Marvis actions this is the self-driving component of Marvis this is where we're

7:23 starting to see customers trust Marvis to actually start issuing trouble tickets and taking action on their

7:28 behalf when they see when Marvis sees problems inside the

7:34 network now when you look at Marvis Marvis has three critical customer

7:39 facing components and we've talked about them we have the conversational interface this will become the new

7:46 interface of choice for most networking teams going forward you know you look at the industry we all grew up with CIS

7:53 2030 years ago we moved to dashboards and I know it's hard to say eventually

7:58 you have to give up on that CLI I know it's like prying you know something from your dead hands but you know we have to

8:04 move on to better ways of interacting because these networks are getting more complicated continuous learning you'll

8:10 see like three new Innovations in this area you know you will talking about adding teams into the mix and we'll be

8:17 talking about bringing a general model to to the party and a new application lle dashboard here and then the action

8:24 framework we'll hear from weson weson soon more about what new actions Marvis is bringing to

8:30 table when you look under the hood um SAR others looking he said Bob no one's

8:35 cares about your little data science models I finally give agree most people do not care about these data science

8:41 models but what you will realize when you look under the hood machine learning is going on under there the thing that

8:48 is differentiating is going to be the Deep learning we saw it with chat GPT

8:53 you will see in networking that these deep learning models are going to start to differentiate what's happening inside of

9:00 networking and when you look at AI what I tell people this is really

9:06 the next step in the evolution of automation right the only thing with this step is we're starting to build

9:14 solutions that can actually do things on par with humans these Solutions are starting to appear to have cognitive

9:20 reasoning you know you see in your chat GPT when you're working it's like how did it do that that's what these deep

9:25 learning these big language models are bringing to the party they start to let you automate in new and different ways

9:31 this is why you may not need to understand the math under it but you should understand what they're capable

9:37 of doing to confirm this includes Rag and Vector database capabilities yeah if

9:43 you look under the hood of we'll go into C I am you'll see a whole Rag and there there's multiple use cases that whole

9:49 large language model stuff is bringing a whole set of new use cases to the party we'll talk about here in a bit very good

9:59 the other thing I always remind people is you know one of the reasons that suj and I left Cisco to start Miss um we

10:07 knew that we wanted to build have a blank sheet of paper We Built This Cloud AI architecture but the other fun reason

10:15 organizationally if you want to solve this Cloud AI Ops problems you have to make sure you get your data science team

10:21 next to your support team your support team is the domain experts they know

10:26 what's going on in these networks they see it every day your data science experts are the ones who understand the

10:32 math and how to apply math to the problem you know and you look and fundamentally when we move into a cloud

10:39 architecture you find out that your support team is really a proxy for your customers because once we have the data

10:46 in the cloud you know everyone has access to the same data the customer the

10:52 support team anyone troubleshooting that is the power of moving to the cloud and you look what happens this is

10:59 my favorite chart you know this is you know stopping when you start this journey and my favorite comment is if a

11:08 vendor is not using its own clay iops and support they have not started the journey to Cloud aops this is the first

11:15 step in the journey we started the journey in 2018 that orange bar shows

11:20 you how long it took me to figure out why people RIS sshing into APS and switches you got to stop it you got to

11:27 get that data back to the cloud you know and that's a cultural change for those who are in the networking business taking away SSH Keys is this

11:35 hard to do for your support team the blue red shows you it's an endless battle right up and down this is a

11:42 week-by-week battle on making sure that the data science and support team are reviewing the latest greatest problems inside of the

11:48 network the a lot of people ask us why does the blue which is the good here that Marvis was able to answer the

11:55 question why does the blue go down and what happens is is as we solve problems

12:01 and Marvis is able to answer questions users are not going to open support tickets with that same problem so

12:07 essentially the way you interpret this is every single week we are we are

12:13 solving newer and newer problem sets and and consequently yeah the the the it

12:19 gets harder and harder and harder because the easier problems are solved ahead in time so Marvis is able to

12:24 answer questions so every single support ticket at Juniper missed 100% of them they first ask the question

12:32 of Marvis and then say hey did Marvis have an answer did it directionally tell me the answer you know did I is the

12:38 answer somewhere in the McLoud is it in the back end of the cloud or did I actually have to get to the AP and you

12:43 could see we still haven't conquered you know any of those vectors in the sense but continually making Improvement this

12:50 is a scorecard for how AI is doing for us this has been an honest to goodness

12:55 the absolute best thing in terms of measurements we've done is me ourselves okay that does not count against my

13:02 minutes yeah how do you judge the that the questions are getting more difficult

13:09 yeah is there some Metric that lets you know yes it's kind of flatlined on the

13:14 blue yeah but things are getting more complex is there some Metric to also know we're not answering easy questions

13:21 anymore so I review so I I we review this with the data science

13:26 and support team once a quarter uh we go go through a list of all the questions we can't answer and then we make pie

13:33 charts of why we can't answer that and what you quickly find out is we're getting to the bottom of the barrel you

13:39 know the the easy hanging fruit those are getting farther we're getting to a tail of hard

13:44 questions so so it it we have categorization for every single ticket there's categorizations of of a a broad

13:51 category and and a microscopic category and then each week we're trying to knock off some of this right the the power of

13:58 the Mis Cloud so we did 15 production pushes major ones last year and that's

14:04 15 opportunities for us to fix and move forward that's that's where a controller based platform or a cloud platform that

14:12 you have to take offline every every quarter to update you know a legacy microservice architecture does not get

14:17 it done right so for us the agility of the cloud actually helps us move forward it's all in the categorization and I

14:23 would say back to the comment about Hardware you know by having our own access points switch and rout there's a

14:29 whole device health element to this where Marvis actually keeps track of the health that's you know a chunk of those

14:36 support tickets are divide into the hardware there's a hardware component that we fixed what so it's uh it's Rocky

14:43 on the Stream So what what are those hard questions what are the things that

14:50 that Marvis can't answer today yeah so so Rocky great question uh so examples

14:56 are where there's a lot of dependency on the customer side you know uh adjacent

15:02 subsystems right so if let's say you know their uh core router is dropping

15:08 packets on a certain VLAN when we switch to the backup data center this happened at one of the largest universities we

15:15 run in the country uh uh from a you know from a complete wireless network

15:20 perspective you know and and that was so far out of the realm of what we touch

15:26 and see and and so that's a question they'll say hey yeah that no Marvis didn't get the answer so it's it's as

15:32 the problem space expands beyond our Subspace that's where it becomes hard yeah uh so when you talk about an answer

15:40 somebody uh is having a problem do they get an answer or is an action also taken by Marv on the infrastructure great

15:46 question so so so if you look at uh to Drew's question how does you know you

15:52 know in in in in the first customer's case we replaced that was Gap 24,000 APS

15:58 from a different V without adding a single extra access point or adding a new cable we've cut

16:04 their you know site visits by 85% service now we cut their you know support incoming support tickets from

16:10 users by 90% how do we do this is it just all marvy's telling you hey you got to take this action that action no we

16:17 are the first and honest goodness true self-driving Network in the industry

16:22 what do that mean what that means is when we see a problem that could be a a

16:28 problem problem that and these are intrinsic to our system that means a software on an AP a hardware on an AP a

16:34 switch or a router where let's say you take a software process if the software process is crashing as Bob said we watch

16:41 the health of every single you know access point every single software process on the access point and if we

16:47 could reset it without waking up it we do I'll give you an example if the ethernet chip on on an AP suddenly was

16:54 doing it was happy and healthy and suddenly see CRC error suddenly see you know spikes of stuff right and then we

17:01 go we say hey I could reset that ethernet chip without waking up the IT team we do if the problem space comes

17:08 back after several tries and it's like hey this thing is coming back we issue a proactive aoma to our customer and say

17:14 hey we have an AP in your ceiling that we won't like back and here's a brand new AP to you proactive R but absolutely

17:20 100% we take actions uh you know remediating software and hardware issues

17:25 intrinsic to our system that accounts for lot of fewer support tickets yeah so what happened in

17:31 the middle you have you you know you see all this you see these you see how you guys have evolved over and over time and

17:37 in the middle you see that oh you started not be able to answer the questions and then this Spike like what

17:43 was going on there every single week is a brand new week it literally could go down next week right it's because the problem space is different that's so so

17:51 we and that's why this is Bob's way of holding ourselves accountable saying don't fool yourselves yeah you have you

17:58 know that middle was 50% yeah it's a harder problem space that we hit yeah you know my theory was the customer

18:04 suppor team is a proxy for our customers if we cannot keep them happy there is no way we're going to keep our customers

18:10 happy have a question yes sir you said that based on you know um

18:17 monitoring what's happening with the access point in terms of resources or any bug is right so who takes the action

18:26 of doing that like is it the team assigned to your account or who does

18:31 that action no no no yeah so uh the the the automated actions I were talking about that's the that's a data science

18:37 model that's running in the cloud and it's in see we can't really nearly keep going rebooting resetting process we got

18:44 it and so one of the things the the boldest statement I I ever make and people will call on it and

18:50 that's okay we want them to be a customer and challenge us Marvis is never wrong why is why how do we make

18:57 that statement because Drive efficacy it takes six weeks to write Corde maybe six weeks to test code we take N9 months to

19:04 drive efficacy into every single Marvis action and so anything that is self-driving it's near 100% efficacy we

19:12 have determined that with extreme high confidence if we take that action we're going to only do good right and so

19:19 that's it's the efficacy that we've built into each of these Marvis actions at the largest our largest customer

19:25 today has 300,000 app access points deployed in one production Cloud you

19:31 know for their Global Network no other cloud in the industry can touch you know even you know 100,000 APS in one

19:37 instance right that customer can tell you Marvis is never wrong right and that's the efficacy we've driven Bob I

19:43 know we they're going to kick us out of keep going let me make up some time here so this is what I call the collaboration

19:51 Canary in your network coal mine you know when your network starts to have

19:56 throughput latency Jitter problems it is going to be your video collaboration users that first experienced that

20:04 problem right and last year what we announced is basically continuous learning where we're taking Zoom data

20:12 right and join it with network feature data from your client from your wireless

20:17 network from your switch Land network and from your sdwan routing Network taking those Network features and

20:24 building a model that can now accurately predict video plus or minus 20

20:31 milliseconds at 90% accuracy now once you have a model that can actually do

20:36 that actually predict something accurately then you can use what we call shapley explainability algorithms to now

20:44 basically give you a ranking feature of what network feature how each Network feature is contributing to that

20:51 prediction right now what we're bringing to the party this year and

20:56 announcing is teens so last year we basically heard everyone say hey that's great but how about teams

21:03 so now we're introducing similarly a model that takes your teams session

21:09 information combines it with your network feature data and predicts your

21:14 Microsoft teams performance level now this will basically allow us to bring the C collaboration Canary Coal Mine

21:22 feature to 90% of our Fortune 500 customers almost 90% of all our customers either have zoom or Microsoft

21:28 teams now as their video collaboration tool the other thing we're bringing to the party is an application SLE page

21:36 right this will now give you direct visibility of how many bad Zoom teams minutes are running on your network and

21:45 it'll quickly start to be at a site level it will start to let you quickly figure out what layer of your network is

21:52 causing post connection Zoom users the most pain is it your client is it your

21:57 wirus link is is it your land link is it your Wan connection this is the problem

22:03 with most of these features is it's not one thing or other it's a combination of things that's causing pain in your

22:08 network there are occasions we'll talk about on the client side where there may be something like a VPN this vpm Ser we

22:15 found that will cause a lot of users pain that this will quickly find the needle and the haystack problem probably

22:21 the most exciting thing that we're bringing to the party this year that that that that bringing that that to

22:27 party um is body call General model what you see here there is no Zoom there is no teams

22:34 data right this is similar to the large language model this is a model that can

22:41 basically predict what the performance of your zoom and teams users will be on

22:47 your network now we actually had a very large retail customer was having a problem we applied this to that Network

22:54 to see what was going on basically informed us that we need to basically take the channel bandwidth from 40 MHz

23:00 down to 20 MHz right because we're seeing a lot of interference That's The Power of a general model that can look

23:06 at a network and start to predict what is a zoom team's experience going to be like on this network this question are

23:12 you are you planning on heading this direction or the reverse you did teams

23:18 and um Zoom were you were you looking at going and hitting all the other SAS providers individually or is the goal to

23:26 go and and go more General role rather than individual hitting each one by itself I'll let Sun talk to the road map

23:34 I would say there are Visions right now where to take this next right you know zoom in teams are probably the big

23:40 applications uh the other thought is to work with app monitoring companies start bringing in a more General look across

23:46 all the apps but but the short answer Keith is uh the most stringent apps on

23:52 our Network or the collaboration apps like voice video apps in fact the problems faced with voice and video apps

23:58 right now is is you know if your CEO had a teams issue everybody's going to jump up and down and go and fix it but if the

24:03 average user in the Enterprise is having a teams issue or Zoom issue we never get to the root cause of it it just passes

24:09 by because these things are transient right so uh we took the hardest one so we can get this data and build truly a

24:17 model that right now we're consuming no joke millions of user minutes on a every

24:24 single day perspective and so we're learning with millions of users minutes the analogy there is large language

24:30 models learned on billions of you know trillions of data points right and so we're learning this and if we could

24:37 crack this it's game over then I don't need to connect to Salesforce and and a SAS applicational service now something

24:44 else we're that's the goal this is building a general model that can truly

24:49 become the game changer how much of this data that you're collecting is from the

24:55 actual application itself so you got Zoom you got teams are you are you

25:01 taking that from the client that's on the device or are you doing some sort of

25:06 API tie in to these applications these are Enterprise customers uh taking it

25:12 directly from the zoom teams API so zooming teams both have apis that keep data per minute or per session on all

25:20 the sessions on the network I think this redate what uh Sadir said you can visualize that if you look at chat GPT

25:27 they built a a very large language model trained on trillions of words of data to predict the next word what we're doing

25:34 here is taking billions video collaboration data points from Zoom team users and building a model that is

25:41 predicting your Zoom team's experience so that is the power of deep learning models right this is what is going to

25:47 trans you know differentiate and transform what's happening in networking so so should it actually I mean do um U

25:54 should we expect to see this uh I mean I'd love to personally see it in Marv's Mini for example I mean I initially

25:59 showed application SES for example right so I know Marvis Min kind of show some

26:05 stuff in there but but I mean I'm I'm visioning and thinking like this showing

26:12 up in Marvis minis because Marvis you know you're you're using I mean we're using your network I mean U you I'm

26:17 going to get there in like Miss devices right so it knows what exactly we are uh uh you know accessing different

26:23 applications right so it can tell me which applications are having issues and problems basically we'll we'll speak to that okay yeah so so this is you know

26:30 roel to your point large language model that ingested words large experience

26:37 model that's ingesting The Experience minutes from these real-time applications first time in the industry

26:43 right so so there's real opportunity here so um anyway so uh um Ali let me

26:49 start with what does this do we are super excited because um you know uh for

26:54 the last seven years in the M dashboard you saw Wireless wired and Van as the three service

27:01 levels that you saw uh with insights and stuff today for the first time we're

27:06 revealing the an application SLE right and what this is doing is it's basically

27:13 you know collating all the user minutes we see at a given site uh or a given

27:18 access point you know at any uh uh level of uh granularity and say okay I had

27:23 4,700 total user minutes on on that site today from a you uh you know uh uh

27:28 collaboration perspective 233 minutes were bad and then you say okay uh you

27:34 know let me go dive into exactly why so this is basically saying let's dissect

27:41 that 233 minutes it says van was a a huge contributor for that and then you go in and say okay now show me more is

27:48 this uh is this a van by the way in this particular example only the Wi-Fi was

27:53 Juniper misk right the wir network the van Network this is the power of the shapley model we can look at so many

28:02 parameters that we don't have to actually directly get this is again if

28:07 you do some research on this Baba and and nav or data scientist introduced us to this thing and it's it's gamechanging

28:13 in that I could say I could say hey this cohort of users are having an issue

28:18 what's commonality for them ah they're all sharing the you know the same vling or they're sharing the same APS in an

28:25 IDF they're sharing the same SS ID same whe and you know maybe they're all on VPN they so it's it's developing cohorts

28:33 and making meaning of those cohorts that that's where this is this is headed and so we're launching this with zoom and

28:40 teams this should be available for you know missed customers as part of their Marv subscription in the next uh you

28:45 know 90 120 days or so um but but brand new SLE will will basically get you down

28:52 to Aha you know uh Jane in finance had an issue with themes no one's going to

28:57 jump up and down and saying ah let me go get down to that particular team's call no I can take you down to you know

29:04 Jane's laptop and say hey what was the Lost latency in Jitter I have an enumeration of every single team call uh

29:12 uh that that uh she's gone through pick an individual team call that she may have complained about and say aha during

29:19 that specific time there was a van issue or maybe client side right so there's there's incredible value in going from

29:26 at a site level at an org level saying what is what's all the that is happening and then taking it down to an individual

29:32 call first time in the industry we're taking all the way down to an individual call and the experience right so now

29:39 it's um it's Rocky again yeah go excuse me um I see clients listed and all of us

29:47 in wifi know that a big percentage of the issue is clients so how deep and how

29:53 much data are you gathering without a client agent in that hey it's the client

30:00 super good question um so on the client side actually what we're doing is we're

30:05 not installing any new agents we're actually leveraging the agent that is the teams agent and the zoom agent and

30:12 we're basically getting from uh zoom and teams Cloud to Cloud so we don't have to

30:17 ask for a new agent to be installed Cloud to cloud and say uh on the client side we get things like what's the CPU

30:25 during the exact team call that was happening so at one of our customers the CPU on certain clients was actually

30:31 contributing to it uh and and how does that correlate for example if I'm downloading a big Microsoft update or a

30:37 Mac OS update during the time I was having this teams call a normal user would not correlate those two things as

30:43 having impact but it does have impact right another thing we get from client Rocky is is is VPN as an example was the

30:51 client using VPN where are they ingressing egressing that kind of stuff sorry go ahead so let's say it's not

30:58 teams and zoom yeah that are the issue how much data are you able to gather and

31:05 see about a client so let's say that it's the you know senior director VP in

31:11 the corner office and their Spotify is flipping right and we get the call how

31:17 much are we able to see from the network side about the client and how much data are you pulling in to see that uh again

31:26 we have a different agent for if they are not teams in Zoom we use our Marvis

31:31 client agent if some if people are are willing to install that we do have that agent that gives us RSSI driver level

31:39 detail driver type detail all of that kind of stuff you know one of our customers has about 40,000 APS they've

31:45 deployed this on 200,000 handhelds our Marvis client and so they they you know

31:51 they are able we're able to get that kind of data but not application data Marvis clients today focus on Layer Two

31:57 what about the so there is a Marvis client you just aren't as much utilizing

32:02 it for teams and zoom or you're enriching we're not installing another agent but for uh if if the option exists

32:10 for and Slava is going to speak to hey um you know uh some of the onboarding stuff we're doing there is an aspiration

32:17 for a grander client that Slava is going to uh you know speak to I won't steal his Thunder but that will also have

32:23 Marvis I'll I interrupt you sorry oh no it's fine I was just so Marvis agent right I know for handheld it is

32:28 available um I know I mean then there was like a Windows agent that correct what's the status and development of the

32:34 windows agent and Mac possibly I think there was an an announcement and and with with iOS 17 there's possibilities

32:41 on Mac uh sorry on iPhone right so Flav will address all of those in the timing of that yeah all

32:46 righty this is Kevin Fran from the BR um you mentioned these are Enterprise

32:52 customers is there any interaction needed to enable this or this just works

32:58 question so Kevin U so they do connect their teams or Zoom they get an we we

33:03 get an API token and and um it's a base subscription on teams and zoom zoom had

33:09 has additional options they don't have to choose those we're connecting with their base Zoom subscription using an

33:15 API token that they would share uh they would input into the M dashboard uh another question um you

33:22 mentioned that some of this visibility is is done done with just the aps if this was a full junip site do you have

33:28 more visibility so so I think the question Kevin uh uh was you know does it need to

33:34 be a full Juniper site the answer is no uh if as long as just Wi-Fi was Juniper missed uh everything you saw uh is is

33:42 still possible no so the question is if it is a full Juniper site are you able

33:48 to uh see more into the network and have more visibility yes yes absolutely absolutely yeah so if there is sdban

33:55 from us uh we get application uh input into this Chaple model if there's switching from us uh um we we we take

34:01 additional input into the model yeah awesome a couple of quick customer stories uh I think we're severely late

34:08 at this point but uh uh but basically uh service now this was a great example this is there uh where the Hyderabad uh

34:16 you know call center essentially uh you know people were some people were complaining specific cohort of people

34:22 were complaining and the shle model said hey these guys their VPN is egressing out of uh Sydney while the users were

34:29 sitting in Hyderabad you know true story and it was it was interesting that nobody actually car to look in that kind

34:34 of format uh Dartmouth College this is a great example uh Brian Ward a lot of you

34:39 guys know uh uh basically this was a corner office the executive has uh you know uh his own uh dedicated AP and so

34:48 you know why would there be a capacity issue life was good and and but the this model was saying hey there's a capacity

34:55 issue and Brian said no I don't think so let's let's duke it out and turns out uh

35:00 what what ultimately was happening was uh you know dmouth runs the largest hotspot uh uh in in higher ed verle they

35:08 have a wide open network and so passer by passing that building constantly connecting through the AP uh from this

35:15 uh executive and that was actually uh challenging so I'll skip some of the example Bob come on up we got to catch

35:21 up on time Marvis minis okay so as a data scientist Marvis minis is probably

35:27 the most exciting announcement of the year because it's bringing data to the party that we did not have before the

35:33 other reason why this is exciting is because similar to the general large experience model you know we are now

35:40 able to predict a user's experience before they actually get on the network the other big thing for me personally

35:46 was making sure this was part of our Cloud architecture and what does that mean that means speed of innovation that

35:53 means we can innovate Marvis minis during our weekly production pushes right this is not part of my firmware

35:59 build this is part of my cloud architecture now secondly we're bringing it to the switch so that also brings

36:05 more data from our switch wir Network into Marvis now of what's going on inside that that Land network and

36:12 thirdly it looks at both the preconnect and postc connect experiences preconnect

36:18 is focused on making sure that authentication dhp DNS service is up and running and going we're also making sure

36:25 critical applications up and running before the door opens and we're bringing a speed test to marvus minis this starts

36:31 to giv us visibility into that Wan connection and throughput connection before the doors open up in the morning

36:38 with that I'm going to turn it back over to Sadir to catch us up let's go did my part quick question on that will we be

36:43 able to customize the speed test possibly I probably want to use a I don't know like a server in my data center or something like that like iper

36:49 or use any not right now not right now it's uh yeah so so for those that may

36:54 not have uh seen or known what Marvis min is let me actually just give you a very quick uh you know view of what this

37:01 is what today Marvis actually does an extremely good job of understanding users experience and and and pinning

37:08 down on what was happening with an individual user but a lot of times I mean if we as the more and more cios we

37:15 speak to their number one uh you know concern and issue is I want to know

37:20 before my users have failure I want to know before anybody experiences failure

37:26 and that's where we launched the the first sort of digital twin uh um uh in in the industry uh from a user

37:32 experience perspective the industry generally when they speak of digital twins they speak of network uh you know

37:38 emulation this is we we've always been focused on the user and this is about

37:44 understanding the users's experience and so being able to actually assert problems before users come on the

37:49 network or experience failure and so what what we are super excited to launch

37:54 today is and and again I think a lot of the Marvis minis piece is um you know

38:00 classic Mist style we want this to be super easy to deploy meaning there is no

38:06 configuration you you deploy missed access points and and you know if you have the Marvy subscription minis are

38:12 just automatically uh you know running what does that mean we learn your ssids we learn your vlans we learn your DHCP

38:19 your DNS your radius this is not you know our grandfather's sensor Network where you have to set up everything or

38:25 if something else changes on the network I to go tweak that this is a modern Cloud native digital twin experience and

38:33 so if a failure happens I actually have a dynamic packet capture waiting for me in the cloud right so so really cool

38:40 stuff well that's great that's what we said you know earlier in January what are we launching today what we're

38:46 launching today is Marvis minis you know is now incorporating you know the

38:53 switching authentication from the switching side so this is step one and I'll get to Oliver you're talking about

38:59 the experience bits so today Marvis minis is basically saying okay I can

39:04 actually a certain you know DHCP DNS radius some level of applications um

39:10 where you know we we learn the the major SAS applications and we are validating this is an example where DCP was failing

39:16 I have a pcap waiting for you now let's take this to um the van speed test today

39:23 we and this is the early days the first version we're launching a basically access points running you know speed

39:31 tests once a once a day just to establish what is the the the maximum bandwidth at this site you know uh and

39:38 you know obviously you can customize this thing and then switches starting to do radius validation every hour to say

39:45 I'm able to reach radius and able to authenticate with radius so again a little bit more data to Bobs point the

39:52 more data the better Marvis is now let's talk about uh where where uh you know

39:57 this could get even more interesting this is brand new that we're launching today where we are now basically saying

40:04 how do we get more visibility into application performance right and so uh

40:10 the idea is is essentially leverage your infrastructure use this digital twinning

40:16 experience to say Okay I want to define or I want to learn you know some of the

40:21 these are my some of the top applications at this site let's say as an example I picked teams as an example

40:26 and say okay I want to know the performance of teams across all my sites

40:31 across L across you know loss latency Jitter so these are all my sites I wanted to know and bucke tize my sites

40:39 for me on what's my average latency across my entire organization and so you

40:45 can immediately get to the outliers on saying which sites are actually having

40:51 persistently you know low lat bad latency or larger latency than others

40:56 right so first gives you an organizational view of site bucketization of application performance

41:03 then you go in and say okay I want to now pick you know a particular site across all my applications I want to

41:09 know in that particular site what is the uh um loss latency Jitter across all

41:15 these applications so now um you know again this is um this is another

41:20 application this is the bucketization of the latency for those applications again no sensors no additional Hardware

41:27 nothing to set up constantly always running and and having the data at your fingertips and so this is getting you

41:35 proactive if something changes at a site where it jumps from you know its normal

41:40 latency to something uh something new we catch it and now I zoom into an individual site here right this is this

41:47 is the same site all five applications that I'm showing they're continuous you know uh experience so constantly we're

41:54 validating application performance uh and this is where I'll leave we are getting closer to you know not requiring

42:00 additional servers and sensors just using the network to validate app performance yeah I like the idea of

42:06 testing the application I know Marvis minis um are initiated from a

42:12 neighboring AP to another one but will you incorporate data because I know you're showing average latency and all

42:19 that will you be able to show us to like the AP it's connecting to and how many clients were connected to that AP cuz I

42:25 I would like to know if that ly is driven by whatever else is going on with the clients Associated good yeah great

42:33 question this is Phase One Step one right lots of possibilities uh you know

42:38 we you know we when we launched Marvis minis in January I you know I genuinely

42:44 feel we have a three-year road map of things that we could do that could truly be gamechanging right so this is step

42:51 one we are laying down the foundations and infrastructure the one difference between January and and and May 24 is

42:58 we've made Marvis minis completely sort of a you know the very first time we've asked our customers to upgrade to a

43:05 certain firmware version from now on Marvis minis is an extension of the cloud new functionality don't have to

43:10 touch firmware it's just a you know a new push from the the cloud type of stuff so we're we're laying down the

43:16 foundation this is the very first phase of it lots of possibilities Yeah you mentioned something about is when you

43:22 mentioned about radius right so is that from the switch but is it also testing like from the access point it does

43:27 access point we already have cuz you know you run into sometimes a scenario for example where um somebody wants to

43:33 add access point IPS in the radius or right like individually or something like that or somehow it got missed or

43:39 something like that it will test that as well access point so so the way Marvis minis so so obviously it's testing all

43:45 the time uh this application performance as an example we're doing this every 5 Seconds we're constantly validating

43:51 everything from a uh um a radius test perspective essentially what we do is

43:57 you know we test and if everything is okay you know you know no harm no foul but if we find one AP failed the radius

44:05 test immediately we start a blast radius determination oh is that one AP is it

44:10 just these APS on the switch is it the APS in this closet is it this floor is it this building that site so we have

44:17 Marvis minis is designed to slowly expand scope to tell you hey all of the

44:22 Eastern Corridor is actually having a radius issue because the clear pass for that thing is down right something like

44:28 that so we we it has the blast radius determination buil and the last question is like will the user will ever be an

44:35 availability where I can maybe possibly add a user and some credentials in there to test against not yet but uh we'll

44:41 we'll get there but today you can instigate a a a test now the test now button actually can go do all of the

44:48 tests uh right now okay yeah awesome all right uh I'll I'll skip some of the uh the the stories on Marv's minis in the

44:54 interested time sorry sorry um I was just going to ask like how is this data like distributed out to uh become

45:01 actionable like like if you know that it's a radius problem can you just set up Marvis or the dashboard to package up

45:07 some screen grabs or reports or data and like send it over to the radius team so they can fix their stuff or the teams

45:13 people so that they can know or the W people so that you're not just digging down into the weeds to like look at

45:18 charts and feeds super yeah but like do something about it correct great question so the way we've actually uh um

45:25 uh done this is all of the Marvis minis um from the rest of the system

45:30 perspective looks like an actual user and so if we determine that hey radius is actually unavailable for this segment

45:37 of the network um that today turns into a Marvis action which which throws a web

45:43 hooks into uh a service now or their automation tool and that that's how it becomes actionable and then we take all

45:49 this data and attach to that right and so yeah so that's how so the consumption not bound for from our system is always

45:56 web hooks through Marv action awesome uh I think again you get

46:01 the stories in the interested time we have one more section uh Bob so so uh you know I'll let you uh speak to

46:09 that so we're going to do a little bit more on conversational interface I think someone brought it up earlier you know

46:15 so this journey as I said started back in 2018 we started this journey with

46:20 natural natural Lang natural language understanding technology back in 2018 we

46:26 started started with realtime troubleshooting questions so we got very good at understanding what people were asking us and being able to respond to

46:33 these realtime try what we announced last year was really around knowledge based with open Ai and chat GPT what

46:40 they brought to the party was natural language generation they gave Marvis a

46:45 voice we're very good at understanding the question now we're getting very good at answering the question like a real

46:52 human and where that's leading to is a much more natural dialogue inter face with Marvis in the future going forward

46:59 when you look under the hood of Marvis what you see is we basically have built

47:05 a platform an architecture that has multiple llm pipelines for multiple use

47:12 cases so depending on what use case you're solving you'll find different open source large language models under

47:18 the hood of Marva C the first use case that we mentioned was real-time troubleshooting right this is where we

47:25 have a query classifi ation algorithm that classify what type of query question is the customer asking us

47:32 realtime trouble shooting goes down One path what we talked about last year was knowledge base right that is where we

47:39 basically started getting much better we've always been able to retrieve documents for user questions we're

47:44 starting to summarize those documents right and that's what chat jpt brought to the party was solving that doc search

47:51 summarization use case and that's what everyone in the industry is working for working on and I would say if there's

47:57 anyone working on this we're right at the Leading Edge please look me up I'm happy to compare notes I think we're all

48:02 in the same boat trying to figure out what's going on with do language models that are changing weekly the other big

48:08 thing we're announcing this year is customer support similar to that real-time efficacy chart you will see an

48:15 efficacy chart for support questions now for knowledge how well are we answering them back to that same theory if

48:22 customer support's not happy customers are not going to be happy and finally what we're looking forward to in the future is starting to put large language

48:29 models on top of SQL this is basically letting our customers start to explore

48:35 their own network data think this is an alternative to bi large language model will become another way to know what's

48:42 going on your network on top of your data and I would say this is the other

48:47 new chart you'll be seeing basically for every knowledge-based question that comes into our support team about half

48:54 of it half the questions that come in realtime troubleshooting the other half are some sort of how do I configure my

49:01 4400 where do I find something these are the questions that llm is going to get

49:07 very good at answering the green and the yellow start to show you how well Marvis is doing at answering those tickets for

49:13 every support ticket that comes into our our Network right now you can see in the scheme of things same thing like other

49:20 one we want to get up around 80% we're probably the blue here represents hey this wrong

49:26 good news is hey it's only 23% so we're off to a good start we want to get that down to 80% going forward hey Bob who is

49:33 asking these questions are these trained it operators that know how to properly

49:39 form a question and know what they're looking for are these first level help desk when I go to talk to Enterprises they're like yeah no we don't we don't

49:45 let our help desk type questions into the into the chat bot in order to get data back they're like heck no we don't

49:50 want them to do any of that and so it it you know if the quality of question is coming from a trained source then that's

49:57 way different than Joe Blow going why is my Wi-Fi broken I would say what we found is a use case what we call bring

50:03 your own question like a uh an SE or a customer asking a general Network question those are relatively easy to

50:10 answer the questions we see coming into this customer support team are usually much harder much more specific you know

50:16 once we once someone issues a support ticket it's usually and it's a range of beginner to expert that means that they

50:23 were not able to find the answer inside the documentation usually to to your point Sam u a lot of them are they they

50:30 haven't spent the time they they're either new to us or new to the industry and they just want to get something done

50:36 fast and the fastest way instead of searching and analyzing is hey support can you tell me how I do this right and

50:43 they could search in our documentation do it themselves they open a ticket today and and and maybe Bob if you click

50:49 one more uh essentially what we're trying to do is uh is essentially get to

50:55 support ticket deflection uh um to to leverage um uh this piece for uh so

51:01 essentially today you know uh you could ask these questions within Marvis people

51:06 can selfs serve uh within the Marvis dashboard ask a certain question but if

51:11 let's say they didn't or they they they didn't know how to or they didn't have Marv subscription or they they didn't

51:18 want to do this you know how do I actually when when I open a support ticket and if I type in hey what the

51:24 hell do these three blinking LEDs mean this is is not a a you know Jan you know

51:30 Finance kind of question this is someone in it in helped ask you know saying hey I got to set this IP up or something and

51:36 and you know that that deflection right now we're at 11% so 11% off our

51:43 questions are getting deflected by we provided the right answer and they

51:48 didn't even open the ticket but I would tell you Sam for anyone who complains about

51:54 documentation this put a big bright light on it I bet it does I bet it does 11 11% is right now we have aspirations

52:01 to do a little bit more but uh is this like an internal CH

52:08 GPT um so we are exposing this to our customers through uh uh through uh through uh the Marvis conversation

52:15 interface but we're going to embed it into everywhere right so Juniper documentation will have an llm uh front

52:22 end to it right our support tickets will have an llm front end to it but we got to get the efficacy uh uh to the right

52:29 place where we feel comfortable to throw people over that because again Hallucination is a real thing with open a eye and chat GPT and and you don't

52:36 want to send people on Wild Goose chases right soini uh let's catch us

52:42 up yeah there you go so sudhir said he can speak fast I will speak even faster I have roughly 30 seconds to cover this

52:48 one minute um but the really cool thing when Bob started the session he talked about

52:55 more telemetry coming into Marvis to always answer the question of what is impacting the end user experience we

53:01 talked about how we are pulling zoom in teams labeled user session data what we are doing with this now again taking it

53:09 to the next level is pulling in Telemetry from the data center because often times you have customers who have

53:14 apps SAS applications as well as data center applications fore an end user

53:20 sitting in the branch and the campus when an issue happens how do you know which layer of the network is the

53:26 problem so with what we've done here what used to be Marvis applications now it says

53:31 Marvis data center applications and now I can actually go from my Marvis ofren

53:37 campus into Marv the Data Center and see exactly what the issues I have in my

53:43 Juniper abstract Control Data Center and pull that back into my branch and campus

53:48 solution so for your network operations team Sam you spoke about the level one

53:53 held desk they now have access to Marvis they can actually see when the ticket comes in oh Wireless is clean wir is

53:59 clean van is clean I have a data center problem let me escalate like Jen like you said escalate the data center team

54:04 to say hey the problem is right here it's impacting these five users in my branch do something go do something but are you're not pulling your data center

54:11 gear into the dashboard you're just asking abstra I'm pulling anomalies from abstra into Market but you're not using

54:17 it for like inventory control and you know configuration and all that stuff that stays in AB stays in Abra so from a

54:23 day two perspect perspective from an aops and to and a stance perspec Ive data center was our next sort of

54:28 building block we're pulling that in into the branch and campus solution so again all about the end user all about

54:34 the end user experience how do we give that seamless view to say where is the issue impacting these end users