{ "segments": [ { "start": 3.32, "end": 6.359999999999999, "text": "Is to be an open question answer session," }, { "start": 6.76, "end": 10.2, "text": "which means to change one of them, there is no set agenda." }, { "start": 10.3599997, "end": 15.32, "text": "Although if no one has questions, I'll find what to talk to you about if if no one is bored yet." }, { "start": 16.434998999999998, "end": 17.634999, "text": "But, also," }, { "start": 17.875, "end": 27.875, "text": "if someone asks a question and you have some feedback or you have a follow-up question on the question you're talking about or you are doing something similar or the same in your environment," }, { "start": 28.035, "end": 38.55, "text": "unmute yourself or put something in the chat. It's supposed to be a discussion, not just question and then me answering and then going forward and so on. Yeah. I'm saying. That's that's" }, { "start": 38.55, "end": 39.51, "text": "the one thing." }, { "start": 39.91, "end": 41.11, "text": "The other thing" }, { "start": 41.829998, "end": 44.79, "text": "is that" }, { "start": 43.625, "end": 53.704998, "text": "just just to keep thing ordered, if your next question is a follow-up on the current one, feel free to unmute. Otherwise, just let the discussion finish before you ask the next one." }, { "start": 54.104996, "end": 64.129997, "text": "And because Teams is good with actually notifying me that you have a raised hand, If you have a question and we're in the middle of another one, just raise a hand so I know you have a question." }, { "start": 64.93, "end": 66.60999699999999, "text": "With that being said," }, { "start": 67.009995, "end": 68.45, "text": "let's kick it off." }, { "start": 68.61, "end": 70.53, "text": "So welcome to the September" }, { "start": 70.93, "end": 71.729996, "text": "ask Annie." }, { "start": 72.545, "end": 78.545, "text": "I will also apologize for my voice if it starts dropping because I'm fighting some floor." }, { "start": 78.545, "end": 80.704994, "text": "I don't know what I managed to caught on my" }, { "start": 81.025, "end": 82.225, "text": "current travels." }, { "start": 83.345, "end": 86.145, "text": "But with that, let's kick it off. Who wants to start?" }, { "start": 88.86, "end": 90.62, "text": "Oh, and I have a live server." }, { "start": 90.94, "end": 99.9, "text": "So if I don't know the answer or if we want to look at something on the server, I can jump a server up on the screen and we can look at the live server." }, { "start": 99.98, "end": 103.1, "text": "So I don't do a lot of PowerPoint on these meetings." }, { "start": 103.715004, "end": 105.635, "text": "So who wants to start it off?" }, { "start": 106.755005, "end": 109.075005, "text": "Joe posted a a question" }, { "start": 109.395004, "end": 114.195, "text": "on Hi. So that one so George is asking to talk about" }, { "start": 115.155, "end": 115.875, "text": "the" }, { "start": 116.115005, "end": 118.035, "text": "new standard cluster. So" }, { "start": 118.69, "end": 122.85, "text": "let's see. I also have a raised hand. So, Hans, let's start with you." }, { "start": 123.729996, "end": 129.729996, "text": "And then Yes. And then Jort will go to the standard cluster and talk about what changed with the Postgre." }, { "start": 129.81, "end": 130.36999500000002, "text": "That okay?" }, { "start": 131.225, "end": 136.98502, "text": "Yes. Fine. Hello, everybody. I asked I sent an email, but I don't know where it is," }, { "start": 137.465, "end": 140.02501, "text": "about, a s two connections." }, { "start": 140.425, "end": 142.42502, "text": "We have new a s two connections." }, { "start": 143.065, "end": 145.93, "text": "And my question question is, is it necessary" }, { "start": 145.93, "end": 148.49, "text": "to create an a s two application" }, { "start": 148.81, "end": 150.49, "text": "with an in and outbox" }, { "start": 150.81, "end": 152.65, "text": "or just make," }, { "start": 153.12999, "end": 154.01, "text": "like," }, { "start": 154.81, "end": 155.93001, "text": "for inbound," }, { "start": 156.09, "end": 157.93, "text": "receive the message just" }, { "start": 158.355, "end": 159.075, "text": "from" }, { "start": 159.155, "end": 161.55501, "text": "a s two transfer site" }, { "start": 162.75499, "end": 163.315, "text": "to," }, { "start": 163.715, "end": 164.19499, "text": "to," }, { "start": 164.75499, "end": 166.11499, "text": "advanced routing" }, { "start": 166.11499, "end": 167.235, "text": "and automatically" }, { "start": 167.235, "end": 169.63501, "text": "receive the files from the transfer site." }, { "start": 170.03499, "end": 171.155, "text": "Correct. So" }, { "start": 171.61, "end": 177.69, "text": "so you probably saw the site mailbox application and that tripped you up. That's an old application." }, { "start": 177.69, "end": 179.53, "text": "You do not need it anymore." }, { "start": 179.69, "end": 183.13, "text": "It used to be the only way to set up a s two, but now" }, { "start": 183.37, "end": 189.54501, "text": "if all you need is to receive, then all you need to do is advanced routing. You don't need to do anything special." }, { "start": 190.425, "end": 193.30501, "text": "Okay. Super. Thank you. That's good. Okay." }, { "start": 193.945, "end": 201.22, "text": "So and this is for this is for everyone. So we carry three old applications, standard router, basic application," }, { "start": 201.3, "end": 203.46, "text": "and a s and the site mailbox." }, { "start": 204.1, "end": 206.5, "text": "While they are working just fine," }, { "start": 206.74, "end": 211.62, "text": "they're also what we call the old way to route. They are the unconditional routing kind of templates." }, { "start": 212.935, "end": 223.015, "text": "So at the moment, the rule is if you can do it with advanced routing, go advanced routing. And there is only one use case that doesn't do that doesn't work with advanced routing," }, { "start": 223.495, "end": 225.895, "text": "and this is part of the standard route." }, { "start": 226.135, "end": 227.33499, "text": "In the standard router," }, { "start": 227.87001, "end": 229.47002, "text": "there is a capability" }, { "start": 230.03, "end": 237.23001, "text": "when the the service account receives the files for them to be prefixed not with the name of the account, but with the subscription ID." }, { "start": 237.71, "end": 243.825, "text": "This is the part and then distribute based on that. This is the part that doesn't work in work in advanced routing." }, { "start": 244.22499, "end": 249.985, "text": "So if you have that use case, you need to use standard route. Anything else, advanced routing can do better." }, { "start": 250.785, "end": 255.3, "text": "Yeah. I so I use advanced routing. Yeah. Good. Okay." }, { "start": 255.54, "end": 258.1, "text": "So Thank very much. Standard absolutely." }, { "start": 258.5, "end": 260.26, "text": "JORT standard cluster." }, { "start": 260.5, "end": 262.5, "text": "Where do you want me to start with that?" }, { "start": 264.82, "end": 265.22, "text": "Well," }, { "start": 266.695, "end": 268.695, "text": "what are the differences between" }, { "start": 268.855, "end": 270.135, "text": "the MariaDB" }, { "start": 270.135, "end": 270.93503, "text": "cluster" }, { "start": 270.93503, "end": 272.295, "text": "and the" }, { "start": 272.535, "end": 274.53503, "text": "Postgres SQL cluster" }, { "start": 275.33502, "end": 276.93503, "text": "and how it reacts?" }, { "start": 277.655, "end": 282.99, "text": "Because right now, we we haven't installed it yet because there was a certificate problem." }, { "start": 283.38998, "end": 286.19, "text": "Mhmm. That's gonna be resolved in the next update," }, { "start": 286.19, "end": 288.03, "text": "the October patch." }, { "start": 288.34998, "end": 290.75, "text": "Okay. We haven't had a chance to look at it yet." }, { "start": 291.22998, "end": 291.94998, "text": "Okay." }, { "start": 292.19, "end": 292.83, "text": "So" }, { "start": 293.87, "end": 298.435, "text": "I will start with the basic meetings, and then we'll go from there if there are more questions." }, { "start": 298.595, "end": 300.435, "text": "The main difference is that" }, { "start": 300.755, "end": 304.115, "text": "we're changing the way the replication works altogether." }, { "start": 304.51498, "end": 306.115, "text": "So in the MariaDB world," }, { "start": 306.54498, "end": 311.345, "text": "you had two MariaDB's on each of the nodes. So let's talk about two node cluster." }, { "start": 311.425, "end": 316.865, "text": "Same applies for the edges, by the way. So I'll just talk about one level, and we can clarify what needed." }, { "start": 317.39999998, "end": 322.76, "text": "And then any server was only working off of its own local database." }, { "start": 323.0, "end": 335.685, "text": "And then every time you made a call, for example, create an account on one of the servers. If it was the primary server, it will write into its own database, then use the RMI call to go to to the servlets to" }, { "start": 336.005001, "end": 340.485, "text": "go to the other server and tell it now write this in your own database," }, { "start": 340.645, "end": 348.489998, "text": "receive back a I got it, and then only then mark the adding as successful in the primary database." }, { "start": 348.81, "end": 351.37, "text": "If the writing was in the secondary database," }, { "start": 351.85, "end": 352.809998, "text": "it would" }, { "start": 353.77, "end": 359.815, "text": "write it to the secondary, sent to the primary, primary the acknowledges, and then the primary sends back again" }, { "start": 359.815, "end": 364.855, "text": "as a distribution because it may have other servers other secondaries." }, { "start": 365.415, "end": 367.655, "text": "So this was for any update in the database." }, { "start": 368.55, "end": 373.03, "text": "Tracking table was only on the primary. As we all know, server locks were always local." }, { "start": 373.59000000000003, "end": 381.59000000000003, "text": "And when a server job had to be done, the primary was picking up who to do it based on what it told the other server is doing." }, { "start": 382.255005, "end": 389.775, "text": "That was the model. It was application level only. The two MariaDB's didn't know about each other in any any way or form," }, { "start": 390.33501, "end": 393.53499999999997, "text": "and each server was working individual individually" }, { "start": 393.53499999999997, "end": 394.975006, "text": "with its own database." }, { "start": 395.98, "end": 397.340004, "text": "With Postgre," }, { "start": 397.420006, "end": 400.380005, "text": "we're basically changing the model completely." }, { "start": 401.100006, "end": 402.22, "text": "First of all," }, { "start": 402.70000500000003, "end": 410.060005, "text": "during a proper operation of the cluster, all the cluster nodes will work off the primary database." }, { "start": 410.925, "end": 411.565, "text": "So" }, { "start": 412.525, "end": 420.045, "text": "both server one and server two will be connected live to the primary database for all of this configuration" }, { "start": 421.485, "end": 430.17, "text": "of anything in the database besides the server logs. So you don't have the primary secondary concept anymore. It's just a primary secondary database." }, { "start": 431.369995, "end": 433.85, "text": "That's one of the things. The replication," }, { "start": 433.85, "end": 448.565, "text": "how the secondary is kept in in on on track is not to the application anymore. Instead, we we set up replications on the database level. So we have a new page. Let me share my screen because I was just looking at that a few seconds ago." }, { "start": 449.445, "end": 451.365, "text": "Let me know when you see my screen." }, { "start": 451.685, "end": 454.8, "text": "I can see it. So that is the September build," }, { "start": 454.96000000000004, "end": 465.12001, "text": "and this is how it looks. It looks very familiar. Right? Because it's basically just says Postgre instead of Maria. But now there is the replication settings here at the end. And if you have a cluster," }, { "start": 465.36, "end": 471.17502, "text": "which I don't have, it's a single one, you can start the database replication, set up a password," }, { "start": 471.33500000000004, "end": 472.53499999999997, "text": "and set up here" }, { "start": 473.095, "end": 476.05501000000004, "text": "what parts of the database will be replicated." }, { "start": 476.13500999999997, "end": 482.2, "text": "If there is a cluster, this will be filled in with the notes and so on. And this is how the replication happens." }, { "start": 482.51999, "end": 489.48, "text": "So that in the live operation, what happens is that server one and server two talk to database one only." }, { "start": 490.51999, "end": 496.095, "text": "It behind the scenes, except for the locks which stay local in the local database. So I'll just" }, { "start": 496.975, "end": 503.21501, "text": "the ST server lock, the whole space just basically doesn't replicate, although it actually can be set up to replicate if needed." }, { "start": 504.735, "end": 505.37502, "text": "When" }, { "start": 505.855, "end": 509.9, "text": "something changes in the database, it's the Postgre application," }, { "start": 509.9, "end": 513.74, "text": "the internal Postgre application that we had weaponized, basically," }, { "start": 513.9, "end": 519.74, "text": "to use to replicate the secondary database. So the secondary database is essentially just sitting there," }, { "start": 520.06, "end": 524.925, "text": "serving only locks when someone touches something. The sec the" }, { "start": 525.645, "end": 529.88501, "text": "because of that, the secondary server actually have connection to both databases," }, { "start": 529.965, "end": 532.205, "text": "but the live connections is only" }, { "start": 532.685, "end": 533.965, "text": "in the primary," }, { "start": 534.58, "end": 539.22, "text": "which gives you on top of everything the queue management of the enterprise cluster now." }, { "start": 539.78, "end": 541.46, "text": "Because in the old world," }, { "start": 542.57999, "end": 553.265, "text": "the queue was managed by the primary, while now each of the nodes actually can pick its own jobs again from the database because we remove the old way to distribute jobs." }, { "start": 553.825, "end": 554.465, "text": "So" }, { "start": 554.625, "end": 559.825, "text": "in some ways, you can think that as a scaled down version of the enterprise cluster," }, { "start": 559.825, "end": 561.265, "text": "but with an embedded database." }, { "start": 563.64, "end": 569.24, "text": "And that's pretty much what it does. Anything we had been explaining how the enterprise cluster is better for" }, { "start": 570.04, "end": 573.72, "text": "distributing jobs now comes into the standard cluster as well." }, { "start": 574.28, "end": 579.07498, "text": "Because of that, there is additional requirement for more users. This is for the replication." }, { "start": 579.5550000000001, "end": 582.915, "text": "There is additional requirement for more ports to be open" }, { "start": 583.155, "end": 592.91498, "text": "so that the databases can talk to each other and so that the secondaries can reach the proper database. In a case of a fail off, if something happens on the primary" }, { "start": 592.84998, "end": 595.4100000000001, "text": "and another server becomes a primary," }, { "start": 595.8899799999999, "end": 603.25, "text": "the database on the current primary becomes the primary for everyone. So everyone in the cluster will join to the other primary," }, { "start": 603.81, "end": 607.16998, "text": "which also means that you don't have sync button anymore" }, { "start": 607.585, "end": 612.1450199999999, "text": "because there is no manual synchronization needed because you have only one valid database." }, { "start": 612.9449999999999, "end": 614.1450199999999, "text": "So the automatic" }, { "start": 614.385, "end": 621.33002, "text": "when the automatic failover happens, everyone will just rejoins to the proper database, and that's it. You don't need to do anything." }, { "start": 621.65002, "end": 626.05002, "text": "If the what old primary shows up and becomes a primary gate," }, { "start": 627.4100000000001, "end": 630.21002, "text": "it will pick up again. So it's just" }, { "start": 630.69, "end": 636.4449999999999, "text": "a totally different model. It will take a little while to get used to it, by the way. But I'm" }, { "start": 636.685, "end": 641.005, "text": "I'm pretty sure that everyone is happy we don't have a manual sync button anymore." }, { "start": 641.8050000000001, "end": 642.525, "text": "Mhmm." }, { "start": 643.565, "end": 647.565, "text": "Yeah. It sounds like it will solve our in progress transactions" }, { "start": 647.8050000000001, "end": 655.6700000000001, "text": "that they keep floating. It's like the only way to get rid of them is to delete the subscription that where it's happening and then recreate it." }, { "start": 655.99002, "end": 662.63498, "text": "And then, you know, then it goes away, but then it floats to another one. So we've we've been having that that problem. And" }, { "start": 663.435, "end": 664.5550000000001, "text": "Yeah. Go ahead." }, { "start": 665.115, "end": 675.3549800000001, "text": "Yeah. In in the you know, it sounds like it's just minor and annoying, but, actually, at some point, it can stop the primary from actually doing anything, delegating anything." }, { "start": 675.76, "end": 684.0, "text": "Yeah. Yeah. Yeah. When the primary is too busy with stuck jobs, it doesn't have any power to send anything to anyone else. In the new model," }, { "start": 684.24, "end": 690.08002, "text": "there is no primary dispatcher anymore. So because everyone is basically on their own, well, not exactly," }, { "start": 690.875, "end": 697.595, "text": "but it also is a lot less likely to kill your secondary by sending it too many heavy jobs while it's busy with app. So" }, { "start": 698.155, "end": 707.1800000000001, "text": "one of the problems with all the old standard cluster is that the primary doesn't really know how busy the secondary is. So if the secondary is on the brink of failing," }, { "start": 707.81998, "end": 727.225, "text": "in an enterprise cluster, the server knows not to pick up more jobs. Right? So it will just stop picking jobs for a while until it can self heal. In the standard cluster, this mechanism didn't work because there was no way for the primary to know that the secondary needs to heal. And when the primary was trying to heal, nothing was working. It was everything was blocked" }, { "start": 727.5450000000001, "end": 729.625, "text": "because there was no dispatcher." }, { "start": 729.625, "end": 730.505, "text": "Now that changes." }, { "start": 731.40002, "end": 740.76, "text": "Part of the reason we did that is because we wanted to eliminate one of the databases. Part of it is we're using a very old library for the dispatcher model that we had to get rid of," }, { "start": 741.08002, "end": 745.1600000000001, "text": "and there was no update for it. And part of it is we know the other model works." }, { "start": 746.0349699999999, "end": 759.63498, "text": "And here is where I will actually warn people something because I don't know because now both servers are going to go against the primary database, this primary database will actually need more power than the old MariaDB database" }, { "start": 760.24, "end": 763.2, "text": "because you have two processing engine connecting to it." }, { "start": 764.24, "end": 764.80002, "text": "So" }, { "start": 765.28, "end": 773.6800000000001, "text": "if someone is planning something and you are low on resources on your primary box, that's the time to start thinking about a little bit more memory and CPU" }, { "start": 773.94498, "end": 777.3050000000001, "text": "just to assist your database and IO and so on," }, { "start": 777.7049999999999, "end": 778.82498, "text": "just as heads up." }, { "start": 779.145, "end": 784.425, "text": "So Is that part of the documentation, you know, installations? Like, well, you know," }, { "start": 786.185, "end": 789.2700199999999, "text": "per files or per per load," }, { "start": 789.2700199999999, "end": 791.35, "text": "you would need this many CPUs," }, { "start": 791.51, "end": 792.63004, "text": "this much RAM?" }, { "start": 793.03, "end": 803.43002, "text": "Well, yes and no. We actually changed the install guide to have a different minimum than it used to be. So we actually updated the minimums to be to something that resembles a production server." }, { "start": 804.125, "end": 805.8050000000001, "text": "So if you follow those," }, { "start": 806.44498, "end": 816.125, "text": "you are more likely to succeed. However, if you are an existing customer and you are on the brink of overflowing your servers and you are riding very close to your capacity," }, { "start": 817.41998, "end": 825.58002, "text": "you probably should update your primary a little bit or your servers a little bit before going into the new cluster just to be on the safe side." }, { "start": 826.38, "end": 827.5, "text": "It drops" }, { "start": 827.5, "end": 831.66, "text": "some of the load that was part of the dispatcher model, but you still have" }, { "start": 831.905, "end": 840.30505, "text": "two servers working on the same database. So, essentially, your primary box now will run an SD server and the whole database for both servers." }, { "start": 841.265, "end": 847.18506, "text": "So You recommend that the Postgres SQL live on the primary server, or can it be" }, { "start": 846.0, "end": 847.52, "text": "Yes. Secondary" }, { "start": 847.52, "end": 859.5199995, "text": "server or it could be a completely different server? So if it is completely different, it's called enterprise cluster now and you need the license for that. It it will live both on the primary and the secondary," }, { "start": 859.5199995, "end": 869.095001, "text": "but it will run off the primary and proper. So remember, you'll still have set up Postgre on each node. It's just that the second one is just getting replicating," }, { "start": 869.095001, "end": 870.615, "text": "waiting for a failover." }, { "start": 873.015, "end": 874.455, "text": "Right? Okay. Yeah." }, { "start": 875.16, "end": 880.76, "text": "It's so so that's the big difference between enterprise cluster and standard cluster that remains." }, { "start": 880.760002, "end": 886.44, "text": "In the enterprise cluster, you have a separate database. So the resources are just for the database on this server," }, { "start": 887.125, "end": 891.605, "text": "while the servers themselves only carry the production the bulk the ST." }, { "start": 892.164997, "end": 897.364998, "text": "While in the standard cluster, it's a database and ST on the same box," }, { "start": 897.605, "end": 900.964996, "text": "and whoever is the primary is getting also connections from the secondary." }, { "start": 901.56, "end": 903.0, "text": "That's the main difference." }, { "start": 904.2, "end": 911.96, "text": "So should make everyone's life a lot easier, especially because now you can see tracking tables from both places. And if someone has some" }, { "start": 912.28, "end": 917.24, "text": "tooling done that is chasing who is the primary so you can read the tracking table, you don't need that anymore." }, { "start": 917.935005, "end": 923.6949999999999, "text": "If you're doing onboarding, it doesn't matter which of the nodes you are onboard into because you don't have the" }, { "start": 923.69501, "end": 925.775, "text": "old rigmarole of" }, { "start": 926.015, "end": 931.375, "text": "either a route on the secondary needs to send to the primary just to be sent back to me and so on. So" }, { "start": 932.21, "end": 933.090004, "text": "efficiency" }, { "start": 933.090004, "end": 937.65, "text": "wise, depending on your use case, we I've seen between 3050%" }, { "start": 937.65, "end": 939.97, "text": "improvement in throughput and speed." }, { "start": 941.81, "end": 945.17, "text": "But this is as long as you don't kill your server, of course." }, { "start": 945.965004, "end": 948.285, "text": "Right? So Right. But" }, { "start": 949.085, "end": 953.965004, "text": "it it's it's a lot more efficient model, and and how much improvement you will see" }, { "start": 954.4449999999999, "end": 962.590004, "text": "in terms of speed and performance and so on really depends on your exact scenarios. There are things that just take time." }, { "start": 962.99, "end": 964.91, "text": "But in terms of reliability," }, { "start": 964.91, "end": 966.51, "text": "it's a lot cleaner model." }, { "start": 966.75, "end": 974.51, "text": "Also, if you lose if the node need to restart because something went horribly wrong, it can self rejoin without even telling you. So" }, { "start": 975.0250100000001, "end": 975.74501, "text": "it's" }, { "start": 975.82501, "end": 977.585, "text": "it's a cleaner model." }, { "start": 978.9449999999999, "end": 985.10501, "text": "And the good news is that it's essentially reusing a lot of the capabilities of the enterprise cluster model," }, { "start": 986.38501, "end": 993.71002, "text": "which means that a lot of the heavy lifting on how to tune that kinks and so on had been done before that. So it's not brand new." }, { "start": 994.11, "end": 994.75002, "text": "So" }, { "start": 995.23001, "end": 999.1500100000001, "text": "that's why we chose one of our databases and not a brand new one." }, { "start": 999.7900099999999, "end": 1000.51001, "text": "K?" }, { "start": 1000.99, "end": 1001.87001, "text": "Alright. Thank you." }, { "start": 1002.725, "end": 1008.40501, "text": "K. Anyone else? Any questions on that or anything else at all? Okay. Miguel." }, { "start": 1009.685, "end": 1020.24, "text": "And I'll apologize again. Me and names. So for anyone I can't get on those ones, I try to read names. I mangle names. I apologize. You tell me how to call you. I try." }, { "start": 1020.56, "end": 1021.20001, "text": "Well, good." }, { "start": 1021.6, "end": 1023.36002, "text": "I think. Okay. So yeah." }, { "start": 1023.84, "end": 1028.16, "text": "Yeah. I have a related question about the same migration" }, { "start": 1028.16, "end": 1029.44, "text": "from Postgres. Sure." }, { "start": 1030.9850000000001, "end": 1032.345, "text": "The server log" }, { "start": 1032.665, "end": 1034.10501, "text": "with MariaDB," }, { "start": 1034.505, "end": 1036.905, "text": "both notes have different logs." }, { "start": 1037.385, "end": 1038.9850000000001, "text": "They are not synchronized." }, { "start": 1038.9850000000001, "end": 1041.38501, "text": "Yes. With pods with Postgres," }, { "start": 1041.785, "end": 1047.84, "text": "we will have a single one or we have two logs? Okay. Several logs remains" }, { "start": 1048.23999, "end": 1049.52, "text": "non synchronized." }, { "start": 1049.92, "end": 1052.8, "text": "Tracking table gets in one place only." }, { "start": 1053.2, "end": 1055.28, "text": "So the logs remain local." }, { "start": 1056.07999, "end": 1056.95999, "text": "Okay." }, { "start": 1056.95999, "end": 1057.76, "text": "Perfect." }, { "start": 1058.23999, "end": 1058.71999, "text": "Thank you." }, { "start": 1061.05501, "end": 1061.77501, "text": "Okay." }, { "start": 1064.575, "end": 1065.85501, "text": "What else do we have?" }, { "start": 1070.175, "end": 1072.815, "text": "I have a big group now and no one wants to talk to me." }, { "start": 1077.25, "end": 1078.45, "text": "Another question." }, { "start": 1078.76999, "end": 1080.05, "text": "Sure. Again." }, { "start": 1080.76999, "end": 1085.33, "text": "If we want to move from the standard cluster to enterprise cluster" }, { "start": 1087.09, "end": 1089.40999, "text": "from Oracle to Postgres," }, { "start": 1090.645, "end": 1091.445, "text": "It's" }, { "start": 1092.005, "end": 1096.005, "text": "procedure to do that, or we have to reinstall everything?" }, { "start": 1096.085, "end": 1097.76501, "text": "What's the best approach" }, { "start": 1098.96501, "end": 1099.845, "text": "in that scenario?" }, { "start": 1102.25998, "end": 1110.18, "text": "You want to move are you enter you're enterprise Oracle now. You want to move to enterprise Postgre or standard Postgre?" }, { "start": 1110.58, "end": 1114.9, "text": "No. At this point, we have a customer with a standard cluster using" }, { "start": 1115.46, "end": 1116.34, "text": "MariaDB." }, { "start": 1117.13998, "end": 1117.53998, "text": "That's right." }, { "start": 1118.2350000000001, "end": 1119.995, "text": "And they want to use too much." }, { "start": 1120.7949800000001, "end": 1121.435, "text": "Yeah." }, { "start": 1121.755, "end": 1125.355, "text": "That's right. Okay. To enterprise, but with Postgres." }, { "start": 1126.155, "end": 1129.195, "text": "Okay. And they want to move to Postgres" }, { "start": 1129.195, "end": 1131.035, "text": "standard or Postgres enterprise?" }, { "start": 1134.57, "end": 1141.53, "text": "I'm not so sure about that. Okay. So okay. So let's explain it like that. If they are going from" }, { "start": 1141.85, "end": 1142.73, "text": "MariaDB" }, { "start": 1142.73, "end": 1156.845, "text": "standard cluster to Postgre standard cluster, to the new modernized cluster we were just talking about, they just need to do the update in place. This will happen for them automatically. They just need to make sure all the prerequisites are done, and this will happen. You don't need to do anything special." }, { "start": 1157.085, "end": 1158.2050199999999, "text": "It's just an update." }, { "start": 1158.845, "end": 1161.64502, "text": "If they want to do to move to enterprise cluster," }, { "start": 1162.31, "end": 1167.35, "text": "my recommendation is to go the path of going to the standard Postgre first." }, { "start": 1167.99, "end": 1171.51, "text": "Because if you look on the screen I was just showing," }, { "start": 1173.1100000000001, "end": 1174.06998, "text": "see at the bottom?" }, { "start": 1174.945, "end": 1178.785, "text": "We actually once you are on this Postgres standard cluster," }, { "start": 1178.865, "end": 1186.945, "text": "we can help move you to an enterprise and move you out. So there is because it's Postgres to Postgres now. Right? Okay." }, { "start": 1186.945, "end": 1188.065, "text": "So there will be a button." }, { "start": 1188.6399999999999, "end": 1195.2, "text": "However, this will mean that they'll need to take an outage until all all of the updates and database copies and what's not happened." }, { "start": 1195.84, "end": 1200.16, "text": "So if they cannot take that, then what they need to do is to reinstall everything." }, { "start": 1200.885, "end": 1202.325, "text": "If they reinstall," }, { "start": 1202.64502, "end": 1208.325, "text": "there are two imports which are important. The XML exporting port, which case account certificates," }, { "start": 1208.325, "end": 1210.40503, "text": "you know, all of the database objects," }, { "start": 1210.725, "end": 1213.36502, "text": "will work from one to the to another." }, { "start": 1214.38, "end": 1219.02002, "text": "So they can export all the accounts, import them on the other side, everything works as expected." }, { "start": 1219.5, "end": 1220.78, "text": "System export," }, { "start": 1221.5, "end": 1223.9, "text": "which is the one on the configuration menu" }, { "start": 1227.18, "end": 1227.98, "text": "over here." }, { "start": 1228.695, "end": 1229.49503, "text": "Oops." }, { "start": 1229.655, "end": 1231.815, "text": "Yeah. My database doesn't like me." }, { "start": 1232.4550199999999, "end": 1234.295, "text": "So the server import," }, { "start": 1234.695, "end": 1241.65503, "text": "the system import export does not work because you have different clusters. So if you have things that are not in the" }, { "start": 1242.3600000000001, "end": 1244.6, "text": "so if you have server configurations" }, { "start": 1244.6, "end": 1253.16, "text": "on the setup menu or on the authentication menu and so on, you'll need to reapply them on the other side. You cannot just move them over." }, { "start": 1253.63998, "end": 1254.2, "text": "So" }, { "start": 1254.6, "end": 1256.52, "text": "that's the only thing to be careful about." }, { "start": 1257.35498, "end": 1260.875, "text": "System export works only with the same" }, { "start": 1261.91498, "end": 1263.435, "text": "server configurations?" }, { "start": 1263.995, "end": 1267.435, "text": "Same server, basically. We call it the same deployment" }, { "start": 1267.835, "end": 1271.835, "text": "because it can be used on the Doctor site as well if you have a Doctor situation." }, { "start": 1272.72998, "end": 1282.00998, "text": "But system export import is only supported as a backup mechanism on the same server. Same configuration, same version, same server, same everything." }, { "start": 1282.33, "end": 1284.97, "text": "Okay. XML import export is portable." }, { "start": 1285.37, "end": 1290.625, "text": "You can get it from one server, move it to another As long as the new server is a" }, { "start": 1291.025, "end": 1294.465, "text": "newer version or the same version, you're good to go." }, { "start": 1295.585, "end": 1296.385, "text": "Okay." }, { "start": 1298.225, "end": 1299.26498, "text": "Perfect. K. Thank you." }, { "start": 1300.58, "end": 1304.1, "text": "Jeff, you had your head up hand up a few seconds ago." }, { "start": 1304.58, "end": 1306.9, "text": "Yeah. So hey, Annie." }, { "start": 1307.1399999999999, "end": 1308.02002, "text": "So" }, { "start": 1309.38, "end": 1309.7, "text": "I" }, { "start": 1310.42, "end": 1313.62, "text": "I've asked them this question knowing that I only have" }, { "start": 1313.925, "end": 1317.445, "text": "about five minutes before I have to jump to a conflict for a little bit." }, { "start": 1319.285, "end": 1323.685, "text": "Is there they kind of in the context of best practices, is there any" }, { "start": 1324.565, "end": 1326.085, "text": "tuning guides," }, { "start": 1326.085, "end": 1328.085, "text": "parameters perhaps" }, { "start": 1327.08, "end": 1327.64, "text": "For" }, { "start": 1328.04, "end": 1329.56, "text": "that are more centered around," }, { "start": 1330.04, "end": 1331.56, "text": "like, the admin UI." }, { "start": 1332.2799997, "end": 1333.3199997, "text": "So, I mean, you'd" }, { "start": 1333.7999997, "end": 1335.0, "text": "you may or may not" }, { "start": 1335.32, "end": 1336.5199995, "text": "they've been aware" }, { "start": 1336.92, "end": 1337.559999, "text": "of" }, { "start": 1338.12, "end": 1338.679999, "text": "some" }, { "start": 1339.08, "end": 1344.025, "text": "of the challenges we've had is, like, the number of routes and and stuff that we have in our environment." }, { "start": 1344.984999, "end": 1347.625, "text": "The admin UI becomes quite slow." }, { "start": 1348.425, "end": 1351.945, "text": "As as you're going in and out of subscriptions," }, { "start": 1352.664999, "end": 1353.625, "text": "routes," }, { "start": 1353.625, "end": 1355.224998, "text": "and stuff, and hitting save," }, { "start": 1355.625, "end": 1359.11, "text": "it's taken, you know, some folks minutes. Yeah." }, { "start": 1359.67, "end": 1360.309998, "text": "So" }, { "start": 1360.55, "end": 1362.47, "text": "you can increase" }, { "start": 1362.47, "end": 1366.149998, "text": "the memory of the admin. Just hit it with more memory," }, { "start": 1366.79, "end": 1371.51, "text": "and you can give more memory to a database. That's pretty much all you can do with admin." }, { "start": 1371.885002, "end": 1375.245, "text": "There isn't that many things we can do with admin, unfortunately." }, { "start": 1376.045, "end": 1376.765, "text": "Okay." }, { "start": 1376.925, "end": 1381.405, "text": "So you said more memory in the admin process and Yes. And okay." }, { "start": 1381.725, "end": 1385.885002, "text": "And the database pulls for the admin just to give it more database connections." }, { "start": 1388.42, "end": 1392.02, "text": "That's pretty much the standard set of things for admins." }, { "start": 1392.26, "end": 1397.939995, "text": "The other thing is we have had historically a few bugs with a lot of the the bigger object sets." }, { "start": 1398.42, "end": 1400.259995, "text": "So if you're not on the latest update," }, { "start": 1401.455, "end": 1403.215, "text": "you might want to check that." }, { "start": 1403.695, "end": 1408.815, "text": "And the usual advice, open ticket with support, tell them exactly where you're seeing the slowness." }, { "start": 1409.215, "end": 1416.415, "text": "Because part of the challenge with r and d is that they literally cannot figure out where to look for those load queries" }, { "start": 1416.630005, "end": 1421.03001, "text": "until we hear from a customer. So whenever you see something going" }, { "start": 1421.19, "end": 1423.990005, "text": "very, very slow, you know, I have 10" }, { "start": 1424.55, "end": 1431.605, "text": "have thousand routes in the same account, and it's slow. Tell us because someone will need to look at that. As much as we're testing," }, { "start": 1431.845, "end": 1437.445, "text": "you know we're not testing with a thousand on a daily basis. Right? No one is clicking any thousand things during QA." }, { "start": 1437.685005, "end": 1439.765, "text": "So we know it works with a thousand," }, { "start": 1440.085, "end": 1448.83, "text": "but no one had done metrics. So that's why we, for example, improved the case where someone has thousands of subscriptions because it was also an operational problem there." }, { "start": 1449.79, "end": 1450.59, "text": "But Gotcha. It" }, { "start": 1451.31, "end": 1460.87502, "text": "just need to know that that that's the reality of it. And it's just a query based. And when you're opening the case, make sure you mention your database because as it turned out," }, { "start": 1461.355, "end": 1467.835, "text": "some of those things are slower on one of the databases than the other just because of how the the database query is crafted." }, { "start": 1468.78, "end": 1470.62, "text": "You know? MSSQL" }, { "start": 1470.62, "end": 1473.82, "text": "and Oracle are vastly different database engines." }, { "start": 1474.37999, "end": 1475.1, "text": "So" }, { "start": 1476.54, "end": 1479.34, "text": "Okay. So I also kinda have a" }, { "start": 1480.21999, "end": 1482.46, "text": "somewhat related question to that," }, { "start": 1483.42, "end": 1484.1399999999999, "text": "which is" }, { "start": 1484.465, "end": 1487.745, "text": "trying to kinda gather metrics in the environment around this." }, { "start": 1488.305, "end": 1490.625, "text": "Outside of us just standing up like" }, { "start": 1491.105, "end": 1497.2649999999999, "text": "browser recorded testing, like a Selenium script to test out some of the UIs to get some baseline of tests," }, { "start": 1499.45, "end": 1502.09, "text": "Is there any kind of mechanism" }, { "start": 1502.09, "end": 1504.97, "text": "or log that you might be aware of" }, { "start": 1505.45, "end": 1507.21, "text": "that kind of helps showcase," }, { "start": 1507.84999, "end": 1508.01, "text": "like" }, { "start": 1510.395, "end": 1511.115, "text": "like a" }, { "start": 1511.835, "end": 1517.75502, "text": "basically, to to grab that metric. Right? Someone goes in, opens a route, hits save. How long does it actually take?" }, { "start": 1518.07501, "end": 1519.11502, "text": "Not really." }, { "start": 1519.675, "end": 1526.35, "text": "Not really. You can see what the audit the admin logs themselves and the Catalina out might have," }, { "start": 1526.67, "end": 1529.71, "text": "but we don't really do any measuring like that." }, { "start": 1531.95, "end": 1533.07, "text": "Gotcha." }, { "start": 1533.15, "end": 1543.21501, "text": "Okay. One more thing. While you are testing, I strongly recommend to run the same update through an API call instead of the UI and see how the speed is" }, { "start": 1543.615, "end": 1544.41501, "text": "because" }, { "start": 1544.49501, "end": 1547.77501, "text": "that helps. So that's usually the first step of troubleshooting." }, { "start": 1547.77501, "end": 1554.3899999999999, "text": "Is the API behind the scenes flow, aka the database query, or is it just the admin not effectively getting the responses?" }, { "start": 1555.19, "end": 1562.3899999999999, "text": "Got it. Because it's a web application on top of a database with an API between them where exactly the problem is need to be isolated." }, { "start": 1562.95, "end": 1568.62502, "text": "So whenever I see a slowness on the API, my first thing is to try the same thing to the API." }, { "start": 1569.585, "end": 1571.505, "text": "That's a really good recommendation." }, { "start": 1571.905, "end": 1578.46501, "text": "Okay. Because And if it was, like, shared infrastructure issues, like database or whatever, you would expect both of them to be slow. Right?" }, { "start": 1579.02501, "end": 1586.78, "text": "Yes and no. Because in a couple of places, especially on the big sets, the database actually doesn't go to the API because it has a faster way to go." }, { "start": 1587.4199800000001, "end": 1594.94, "text": "But if both of them are slow, it's obviously the database. If one of them is faster, it it just helps analyzing the situation." }, { "start": 1595.1, "end": 1597.4199800000001, "text": "Because I can guarantee you, if you go to" }, { "start": 1598.045, "end": 1601.725, "text": "r and d or support from their r and d with this question," }, { "start": 1601.965, "end": 1605.885, "text": "that's one of the things they want you to try anyway. So you might as well try." }, { "start": 1606.365, "end": 1614.48, "text": "Yeah. Yeah. And, definitely, it's something to gather metrics for too. So Yeah. And it also makes it with APIs, it's a lot easier to get metrics from." }, { "start": 1614.88, "end": 1615.6, "text": "Yeah." }, { "start": 1616.48, "end": 1622.56, "text": "Which, again, doesn't help. And that's the other thing. Remember that we are working the so if it is about the routes," }, { "start": 1623.145, "end": 1626.185, "text": "can you try the new better routes menu?" }, { "start": 1630.9850000000001, "end": 1632.745, "text": "I don't believe so." }, { "start": 1633.145, "end": 1635.865, "text": "And I don't believe so because I don't believe" }, { "start": 1636.1399999999999, "end": 1642.1399999999999, "text": "I forget what version we're on, but I think the better routes UI, that only just recently came out. Right?" }, { "start": 1642.54, "end": 1646.7, "text": "It had been coming out in the last year or so slowly." }, { "start": 1647.18, "end": 1653.785, "text": "So if you have it, so the does every version has more and more pieces? Even if you are not able to do everything," }, { "start": 1654.025, "end": 1660.105, "text": "whenever you have a slowness in the main menu, check if the beta is there in the whatever you're doing is available" }, { "start": 1660.66498, "end": 1660.8249799999999, "text": "Mhmm." }, { "start": 1661.49, "end": 1663.57, "text": "And see if it doesn't work differently." }, { "start": 1663.8899999999999, "end": 1664.85, "text": "Because it's" }, { "start": 1666.0500200000001, "end": 1673.8899999999999, "text": "it uses the same calls behind the scenes, but the UI component, the ones that sometimes are the delayers are different. So that's" }, { "start": 1675.13498, "end": 1680.895, "text": "you know, just as a case, it's not ready to be used yet because there are missing pieces." }, { "start": 1681.2949800000001, "end": 1687.13498, "text": "Mhmm. But if you have it, it's a good test. If you're seeing very slow big slowness in the route packages" }, { "start": 1687.57, "end": 1690.85, "text": "to see if the beta is behaving people are better in your use case." }, { "start": 1691.97, "end": 1693.81, "text": "Okay. So I know particularly" }, { "start": 1693.81, "end": 1696.0500200000001, "text": "the slowness the the screens" }, { "start": 1696.21002, "end": 1698.77002, "text": "we've seen folks have slowness on is" }, { "start": 1698.93, "end": 1702.61002, "text": "under the accounts in the routes and subscriptions. That one shot." }, { "start": 1703.39502, "end": 1704.915, "text": "Yeah. For that one," }, { "start": 1705.23502, "end": 1726.1, "text": "increase I don't think there's a beta UI for that yet. Nope. Right? No. No. We started with the templates. Okay. In which case Got it. Check the API as well, and as I said, talk to r and d, but to talk to support. Part of the challenge with this specific UI over there is that in order for us to be able to show you what we want to show you, we literally send the whole account object out" }, { "start": 1726.66, "end": 1727.78003, "text": "in the request." }, { "start": 1728.1, "end": 1744.14502, "text": "And when you have, you know, 100 routes and 10,000 subscriptions or whatever volumes you have these days, it's a lot of objects to run through. And we're doing a lot of additional queries all over the place, and it works beautifully and very fast if you get five of them. Not that fast on a 100." }, { "start": 1744.79092, "end": 1746.79095, "text": "So Oh, right. Right." }, { "start": 1747.2709399999999, "end": 1748.07095, "text": "Okay." }, { "start": 1748.15094, "end": 1749.03094, "text": "That answered my question." }, { "start": 1749.0, "end": 1751.3999999, "text": "Thanks, Annie. Okay. Good." }, { "start": 1751.48, "end": 1755.64, "text": "Okay. I have a question a couple of questions in the chat." }, { "start": 1757.88, "end": 1758.92, "text": "Performers," }, { "start": 1758.92, "end": 1761.559999, "text": "what is the best part? So that's Brian, Justin," }, { "start": 1762.12, "end": 1762.679999, "text": "I think." }, { "start": 1763.415, "end": 1765.815, "text": "Sorry if I missed the" }, { "start": 1765.815, "end": 1766.695, "text": "the order." }, { "start": 1769.095, "end": 1774.775, "text": "The best practice around how much compute resources should be allocated to the server based on the load." }, { "start": 1777.76, "end": 1780.08, "text": "So this is a little bit like the" }, { "start": 1780.800001, "end": 1782.08, "text": "egg and" }, { "start": 1782.96, "end": 1784.48, "text": "chicken question" }, { "start": 1785.2, "end": 1786.16, "text": "because" }, { "start": 1786.8, "end": 1791.84, "text": "load for one person is different for another. So we do have the capacity guide," }, { "start": 1792.575, "end": 1799.855, "text": "which has a very nice table showing key what exactly we had been testing with to get specific results" }, { "start": 1800.575, "end": 1802.575, "text": "from where we can interpret." }, { "start": 1803.295002, "end": 1808.83, "text": "What we also know is that there are operations on the server, for example, PGP," }, { "start": 1809.07, "end": 1811.47, "text": "which are very CPU and memory," }, { "start": 1814.11, "end": 1814.99, "text": "exhaustive." }, { "start": 1815.07, "end": 1817.87, "text": "So depending on what you are doing exactly," }, { "start": 1819.275, "end": 1822.235, "text": "you might need to play with that." }, { "start": 1822.555, "end": 1825.675, "text": "So the way I would start when I'm doing" }, { "start": 1827.115005, "end": 1827.995, "text": "resources" }, { "start": 1827.995, "end": 1831.435005, "text": "is, first of all, as I mentioned, the install guide was updated." }, { "start": 1831.97, "end": 1836.29, "text": "So we have a new minimum requirement for both CPUs and memory." }, { "start": 1836.53, "end": 1842.85, "text": "For example, on the server, the absolute minimum will require now is four CPUs and 60 gigabytes of memory," }, { "start": 1843.09, "end": 1846.61, "text": "but this is for a very slow small systems. Still production, but small." }, { "start": 1849.115005, "end": 1851.435005, "text": "As long as if you are in a proper" }, { "start": 1851.755005, "end": 1855.195, "text": "production environment, I would usually recommend doubling that." }, { "start": 1856.235, "end": 1858.475006, "text": "So four to eight CPUs" }, { "start": 1858.475006, "end": 1859.995, "text": "with" }, { "start": 1859.37, "end": 1865.29, "text": "32 gigabyte of memory is a great start for pretty much any environment that is not on the very small scale." }, { "start": 1865.69, "end": 1869.53001, "text": "Obviously, if you're going to go million, like, million files per day," }, { "start": 1870.170006, "end": 1871.93, "text": "then you need to go a little larger." }, { "start": 1873.525, "end": 1874.485, "text": "So it's" }, { "start": 1876.005005, "end": 1877.60501, "text": "that's where it starts." }, { "start": 1877.685, "end": 1878.32501, "text": "So" }, { "start": 1878.565, "end": 1883.525, "text": "do you have what kind of transformations are you doing? So what happens to your files when they are coming?" }, { "start": 1885.56999, "end": 1890.93001, "text": "Brian? Yeah. That's perfect. And I'm also I'm also on the same team as Jeff at Wells Fargo." }, { "start": 1891.41, "end": 1894.05, "text": "Okay. And so, yeah, essentially," }, { "start": 1894.05, "end": 1896.13, "text": "there's there's all sorts of, you know," }, { "start": 1896.45, "end": 1902.225, "text": "just like you said, different types of loads, which is great to account into the consideration," }, { "start": 1903.025, "end": 1926.2050199999999, "text": "which I wasn't accounting for. So that's great. And I'll definitely have to keep an eye on that. And I really do appreciate that you noted that that stuff has been updated in the install guide. So I'll be sure to go ahead and check those out for further details, but that's that's a great idea and a great starting point for us. So thank you. Yeah. And because I happen to know your environment very well, all of them, actually," }, { "start": 1927.80501, "end": 1931.88501, "text": "just read whatever and then double it and then start thinking again." }, { "start": 1932.205, "end": 1934.52501, "text": "It's my recommendation for your environment." }, { "start": 1934.845, "end": 1937.40501, "text": "You're on the bigger scale, always had been." }, { "start": 1938.205, "end": 1938.52501, "text": "So" }, { "start": 1940.65001, "end": 1946.65001, "text": "other from that, a couple of things to remind to mention, especially if you're building new environments," }, { "start": 1946.65001, "end": 1947.77002, "text": "do baselines." }, { "start": 1948.09001, "end": 1951.93001, "text": "Figure out how long something takes to the server and run it carefully" }, { "start": 1951.93001, "end": 1954.65001, "text": "or at least once a day to see if it's still behaving." }, { "start": 1955.55501, "end": 1964.51501, "text": "You you will never know if the server is getting overloaded if you don't have a baseline to compare to, and this bass baseline need to be cleaned. Right?" }, { "start": 1965.475, "end": 1973.09001, "text": "You cannot just compare one file today to another tomorrow. They have nothing to do with it in terms of where it's coming from or where it's going to." }, { "start": 1974.29001, "end": 1975.41002, "text": "The other thing," }, { "start": 1975.8899999999999, "end": 1977.73001, "text": "in your kind of environments," }, { "start": 1978.85, "end": 1982.21, "text": "and this is very large environment, but there's also it's valid everywhere." }, { "start": 1982.55501, "end": 1983.835, "text": "For the last" }, { "start": 1984.07501, "end": 1986.71501, "text": "maybe ten years, in my experience," }, { "start": 1987.27501, "end": 1988.475, "text": "the biggest" }, { "start": 1988.55501, "end": 1989.5149999999999, "text": "bottleneck" }, { "start": 1989.5149999999999, "end": 1993.67502, "text": "always ends up being the file system we're putting the files onto." }, { "start": 1993.99501, "end": 1995.595, "text": "It's not ST itself" }, { "start": 1996.79001, "end": 2003.03001, "text": "because we had been increasing the capacity. It's usually the NFS or whatever you you are using behind it." }, { "start": 2003.59001, "end": 2011.19, "text": "So or the network separation point or something similar like that, something outside of ST. Because people tend to" }, { "start": 2011.675, "end": 2018.475, "text": "make their ST very sturdy and make it a huge application and then forget that we actually need to talk to the file system," }, { "start": 2018.875, "end": 2020.715, "text": "and we need a network in and out," }, { "start": 2021.115, "end": 2025.35498, "text": "and we need access to the LDAP if you are doing LDAP for authentication and things like that." }, { "start": 2026.37, "end": 2030.37, "text": "So whenever you or if you use iCAP for antivirus," }, { "start": 2030.37, "end": 2033.41, "text": "if your iCAP engine can only handle three files per hour," }, { "start": 2034.0500200000001, "end": 2039.97, "text": "I have some news for you. Your OST will not be able to handle more either because we need to wait on them. Right?" }, { "start": 2041.165, "end": 2041.80502, "text": "So" }, { "start": 2042.125, "end": 2043.085, "text": "Absolutely." }, { "start": 2043.165, "end": 2045.405, "text": "Things like that. Things outside." }, { "start": 2046.045, "end": 2047.96503, "text": "And talking about that," }, { "start": 2048.365, "end": 2051.885, "text": "the one thing that I would point out is archiving." }, { "start": 2051.885, "end": 2054.445, "text": "So we did a huge amount of work" }, { "start": 2054.765, "end": 2055.325, "text": "on" }, { "start": 2055.7200000000003, "end": 2058.04, "text": "getting our archiving" }, { "start": 2059.08002, "end": 2060.52002, "text": "to work a little better." }, { "start": 2061.64, "end": 2063.96002, "text": "For the ones that are not familiar," }, { "start": 2063.96002, "end": 2067.08002, "text": "archiving is a feature you can enable in ST," }, { "start": 2067.32, "end": 2074.1549999999997, "text": "which while we're writing the file on the file system, we write a second copy in the archive folder, and this is what is used for REST submit." }, { "start": 2074.63498, "end": 2082.79498, "text": "So it's a very nice thing to turn on. But until half a year ago, we recommended big environments to actually disable it because it was a hack on resources." }, { "start": 2083.31, "end": 2084.67, "text": "We finally fixed that." }, { "start": 2085.23, "end": 2089.79, "text": "So now it's we actually recommend it to be enabled till now you can have resubmit cleanly." }, { "start": 2089.87003, "end": 2096.99002, "text": "But please, please, please don't put the archive folder on the same shared storage where your file home folders are," }, { "start": 2097.615, "end": 2104.255, "text": "because I've seen people doing that. First of all, archive in the same folder as the actual storage makes no sense whatsoever," }, { "start": 2104.495, "end": 2105.775, "text": "operation.net." }, { "start": 2106.175, "end": 2115.14, "text": "But the bigger problem is that now we have a double IO operation during the receiving of the file going on the same disk that is already struggling." }, { "start": 2115.53998, "end": 2116.74, "text": "So don't do that." }, { "start": 2117.38, "end": 2124.41998, "text": "And just as a base rule, if you have two storages and one of them is better than the other in terms of resources, IO, memory, you know," }, { "start": 2125.145, "end": 2130.745, "text": "use the better one for home folders, the worst one for archiving. Archiving has a single write" }, { "start": 2130.98502, "end": 2138.025, "text": "during the receiving of the file and a single read if and it's only happening a read only if there is a resubmit needed." }, { "start": 2139.9700000000003, "end": 2140.69, "text": "So" }, { "start": 2141.17, "end": 2142.13, "text": "makes sense?" }, { "start": 2144.61002, "end": 2149.41, "text": "Graham? Yeah. That's excellent. Thank you so much. Okay. Talking about resubmit," }, { "start": 2149.89, "end": 2156.4049999999997, "text": "how many people saw in the August release that we have a multiple resubmit through the UI." }, { "start": 2157.365, "end": 2167.125, "text": "My new server died, so I cannot show you. But you don't if you have more than one resubmit to be done, we actually get the and it's it's not an API based," }, { "start": 2167.285, "end": 2178.13, "text": "unfortunately. They're looking into it. But as long as you can get them on the same page on the UI, you can hit a single resubmit button and resubmit multiple transfers at the same time now." }, { "start": 2181.33, "end": 2183.56998, "text": "No one cares. Okay. I had mentioned it." }, { "start": 2185.385, "end": 2186.105, "text": "Okay." }, { "start": 2186.98502, "end": 2188.42502, "text": "Anything else, Brian?" }, { "start": 2190.665, "end": 2192.825, "text": "No. That's it for me. Thank you so much." }, { "start": 2194.665, "end": 2195.305, "text": "Absolutely." }, { "start": 2197.19, "end": 2201.1900299999998, "text": "Okay. Let me give me a second to see just something." }, { "start": 2201.51, "end": 2204.79, "text": "And I saw the other question is about the port changes." }, { "start": 2205.91, "end": 2209.03003, "text": "The release notes has the exact requirements" }, { "start": 2209.03003, "end": 2210.55002, "text": "for the disk to be successful," }, { "start": 2210.88498, "end": 2217.125, "text": "and it has a couple of links as well. So I'll start start with the September build release notes, Jacob." }, { "start": 2220.005, "end": 2220.885, "text": "Okay." }, { "start": 2222.965, "end": 2224.16498, "text": "I'll take a look at that." }, { "start": 2224.87, "end": 2230.87, "text": "Yep. And, also, we published somewhere on the support site an article explaining some of the differences." }, { "start": 2231.43, "end": 2238.79, "text": "I don't have a handy link at the moment. If you cannot find it, I'll see if I can find it so we can I we can send it later, but it's pretty straightforward?" }, { "start": 2239.875, "end": 2241.23502, "text": "From your perspective," }, { "start": 2241.715, "end": 2248.275, "text": "as so if you look through the release notes, it will tell you you need to create additional OS level user, for example," }, { "start": 2248.755, "end": 2255.84, "text": "for the replication to work and things like that. But it is listed as the prerequisite for the update on the September release." }, { "start": 2256.56, "end": 2257.76, "text": "And it's very," }, { "start": 2258.15997, "end": 2263.92, "text": "very clearly spelling out if you're on Windows, if you're on appliance, if you're on Linux, do this, this, this." }, { "start": 2265.69696, "end": 2266.25696, "text": "So," }, { "start": 2266.57697, "end": 2268.337, "text": "it it's pretty straightforward," }, { "start": 2268.337, "end": 2270.57697, "text": "at least which ports need to be opened," }, { "start": 2270.897, "end": 2272.0969999999998, "text": "who needs to talk to whom." }, { "start": 2272.0, "end": 2272.96, "text": "And so on." }, { "start": 2276.08, "end": 2279.6, "text": "And don't forget that this also applies for the edges" }, { "start": 2280.16, "end": 2291.435, "text": "because we're changing the model, but we're also replacing fully the database. So when you update your even if your enterprise cluster, if you update to the September or later release, your edges" }, { "start": 2291.675, "end": 2293.355, "text": "will now be on Postgre." }, { "start": 2293.515, "end": 2295.355, "text": "So if you have replicating" }, { "start": 2295.355, "end": 2296.154999, "text": "edges," }, { "start": 2296.875, "end": 2300.235, "text": "which means you had to enable the synchronization between them or the replication," }, { "start": 2300.56, "end": 2304.48, "text": "They will follow the same model as the sec as the servers." }, { "start": 2304.64, "end": 2309.76, "text": "Everyone uses the database on the primary edge. If they are nonreplicating," }, { "start": 2309.92, "end": 2312.079998, "text": "then, obviously, they they stay still." }, { "start": 2313.085003, "end": 2317.085, "text": "So and but here is also the conversation to be had if" }, { "start": 2317.405003, "end": 2321.645, "text": "it makes more sense now to make them replicating given if they were not before." }, { "start": 2322.045002, "end": 2325.325, "text": "So there are some choices to be made on the edges, but you have time for that." }, { "start": 2325.9, "end": 2335.18, "text": "But don't forget that September or later, we'll change the database on the edges, and you need to follow the guide, and it's not as straightforward as run the double installer." }, { "start": 2335.5, "end": 2339.5, "text": "The September installer is also a double. So we first" }, { "start": 2338.975, "end": 2343.935, "text": "update the the installer and then the ST as we sometimes do." }, { "start": 2344.415, "end": 2345.935, "text": "But the prerequisites" }, { "start": 2345.935, "end": 2352.095, "text": "are pretty severe because it they require a new OS user and a couple more things. So just heads up." }, { "start": 2352.94, "end": 2357.420006, "text": "Read your release me. It's always a good thing to read the release file," }, { "start": 2358.86, "end": 2360.3, "text": "but this time is crucial." }, { "start": 2360.54, "end": 2362.700005, "text": "If you don't, your update will fail." }, { "start": 2363.58, "end": 2364.86, "text": "Don't ask me how I know." }, { "start": 2366.885, "end": 2372.245, "text": "Okay. So I was reading through the read me file, and I I didn't see anything about the ports" }, { "start": 2372.245, "end": 2374.485, "text": "or any of the other package changes." }, { "start": 2378.520004, "end": 2383.8, "text": "Hold on a second. Let me just open it very quickly and see if I still don't have it open, by the way." }, { "start": 2386.600006, "end": 2389.4, "text": "Okay. I have it open. Let me see what it says." }, { "start": 2390.36, "end": 2391.8, "text": "Free space required," }, { "start": 2391.8, "end": 2393.48, "text": "temporary space required." }, { "start": 2396.015, "end": 2397.295, "text": "Two" }, { "start": 2397.615, "end": 2398.815, "text": "OS packages" }, { "start": 2398.975, "end": 2402.895, "text": "need need to be updated, libxml two and libxsd." }, { "start": 2403.295, "end": 2406.095, "text": "The POS will require additional non root user to operate." }, { "start": 2407.90001, "end": 2411.18001, "text": "And you are right. The database ports are not here." }, { "start": 2412.46, "end": 2413.18001, "text": "Okay." }, { "start": 2413.34, "end": 2415.82, "text": "Let me go and find the article then." }, { "start": 2416.38, "end": 2417.18001, "text": "Give me a second." }, { "start": 2424.635, "end": 2425.435, "text": "Kick" }, { "start": 2425.435, "end": 2425.83499, "text": "me out." }, { "start": 2473.08, "end": 2475.535, "text": "I'm kind of blind with" }, { "start": 2475.695, "end": 2477.85501, "text": "Teams. Oh, yeah. I know. I" }, { "start": 2478.175, "end": 2479.37502, "text": "know. I'm sorry." }, { "start": 2479.775, "end": 2487.135, "text": "No. No. No worries. But I was wondering if you were sharing something. I No. I heard that you were kicked off, but" }, { "start": 2489.07, "end": 2490.91, "text": "Yeah. You know what?" }, { "start": 2491.47, "end": 2499.07, "text": "Picked out, not kicked off. No. That's okay. It was just me looking on another screen for a second here. I apologize." }, { "start": 2500.67, "end": 2507.865, "text": "Jacob, let me find this article because I know I saw it somewhere. I just don't reme oh, it's in the admin guide." }, { "start": 2508.985, "end": 2515.22499, "text": "Look at the new install guide and new admin guide. That's the answer, actually. We updated the documentation. That's why I cannot find an article." }, { "start": 2516.38, "end": 2520.3, "text": "So because starting from September, this is the new land," }, { "start": 2520.45999, "end": 2522.46, "text": "the the the new rule," }, { "start": 2522.94, "end": 2523.81999, "text": "it basically" }, { "start": 2523.98, "end": 2533.165, "text": "the new requirements are listed in the two guides. The install guide for initial setup and then the admin guide that describes everything. We replace the old information with the new information." }, { "start": 2535.005, "end": 2537.80502, "text": "So let's see if I can bring that up." }, { "start": 2541.085, "end": 2542.20502, "text": "Docs, axe, wait." }, { "start": 2568.615, "end": 2570.85498, "text": "Let me share my screen. Hold on." }, { "start": 2573.38, "end": 2574.82, "text": "Share. Share. Share." }, { "start": 2579.06003, "end": 2580.34003, "text": "Can you see my" }, { "start": 2582.26, "end": 2582.98, "text": "screen?" }, { "start": 2583.2200000000003, "end": 2586.90002, "text": "Yep. I do. So starting with the getting started," }, { "start": 2591.295, "end": 2592.8150299999998, "text": "initial configuration," }, { "start": 2593.855, "end": 2595.295, "text": "server checklist." }, { "start": 2597.135, "end": 2601.45502, "text": "I think all of the ports should have been updated over here as my" }, { "start": 2602.53998, "end": 2603.66, "text": "post gray." }, { "start": 2607.41998, "end": 2608.46, "text": "Where" }, { "start": 2608.62, "end": 2610.13998, "text": "is my install right now?" }, { "start": 2611.65997, "end": 2612.3, "text": "Hold on." }, { "start": 2631.5299999999997, "end": 2632.16998, "text": "Right." }, { "start": 2632.9700000000003, "end": 2633.77, "text": "Okay." }, { "start": 2637.13, "end": 2640.01, "text": "It's not my regular browser, unfortunately." }, { "start": 2640.09, "end": 2643.21, "text": "So as you can see, I'm still going a little bit to find what I need." }, { "start": 2644.32498, "end": 2647.525, "text": "Give me a second. Guys, let me just" }, { "start": 2670.2, "end": 2671.639999, "text": "So prerequisites" }, { "start": 2671.639999, "end": 2672.599999, "text": "for Linux." }, { "start": 2675.964999, "end": 2681.884998, "text": "And over here, things had been updated completely now with the for the new databases." }, { "start": 2684.204998, "end": 2685.645, "text": "Was 30 environments," }, { "start": 2687.759998, "end": 2689.279999, "text": "post grad databases." }, { "start": 2691.759998, "end": 2694.079998, "text": "This is for enterprise clusters," }, { "start": 2694.32, "end": 2697.199997, "text": "and then the admin guide will have something. So" }, { "start": 2698.0, "end": 2702.72, "text": "which what you're looking for which is which ports we're using for the conversation now. Right?" }, { "start": 2704.455, "end": 2706.055, "text": "Yeah. Which ports" }, { "start": 2706.295002, "end": 2706.855, "text": "and" }, { "start": 2708.215, "end": 2709.095, "text": "potentially" }, { "start": 2710.615, "end": 2714.935, "text": "the the the the article that I have that I've been going off of" }, { "start": 2715.735, "end": 2716.775, "text": "has" }, { "start": 2720.27, "end": 2721.79, "text": "it has the ports" }, { "start": 2722.11, "end": 2724.51, "text": "and it says the new packages." }, { "start": 2725.79, "end": 2727.15, "text": "It doesn't say which Oh," }, { "start": 2728.51, "end": 2733.229996, "text": "yeah. These are the two from the release guide. So the release guide have the two packages required." }, { "start": 2733.805, "end": 2747.244995, "text": "If you look, it's it's a couple of lines into the note over there. The there are just two of them. If you are coming from a new enough version of MariaDB, so if you're in the last three months, we already updated before the the the postgre requirement." }, { "start": 2748.59, "end": 2755.78999, "text": "But if you're coming from a lot older, they're in there. But, basically, if you look at the screen over here on the requirement for installing on Linux," }, { "start": 2756.42999, "end": 2759.549995, "text": "those are the ones that we require require" }, { "start": 2759.78999, "end": 2763.149994, "text": "and the two that are really required for Postgre disguise." }, { "start": 2767.865, "end": 2768.985, "text": "The LeapXML" }, { "start": 2768.985, "end": 2770.825, "text": "two and the LeapX SLT." }, { "start": 2771.305, "end": 2777.945, "text": "Gotcha. So what happened when we dropped the September release is that we updated the admin guide and the" }, { "start": 2778.29, "end": 2781.89, "text": "install guide to be for the September release specifically now." }, { "start": 2782.21, "end": 2798.24501, "text": "Although in a couple of places, there are notes if you're on older versions, you need to do that and so on. So see how for The US user route. For example, now we have this line for the PostgresQL requires a new new non route user. This is also in the release notes. So if you're starting brand new," }, { "start": 2799.285, "end": 2803.60501, "text": "you will already know that. But if you're coming from an old one, that's why it's in the update." }, { "start": 2804.005, "end": 2804.645, "text": "Makes sense?" }, { "start": 2805.44, "end": 2806.16, "text": "Okay." }, { "start": 2807.76, "end": 2810.24, "text": "And then on a semi related" }, { "start": 2810.4, "end": 2810.8, "text": "note" }, { "start": 2811.28, "end": 2812.0, "text": "Mhmm. One" }, { "start": 2812.56, "end": 2816.08, "text": "of the ports that's no longer required is the four four four port." }, { "start": 2816.96, "end": 2818.64, "text": "Between the two servers only." }, { "start": 2820.19499, "end": 2821.075, "text": "What was that?" }, { "start": 2821.635, "end": 2827.155, "text": "Oh, between the two servers. Gotcha. You still need it to run because that's where your admin UI runs." }, { "start": 2827.39499, "end": 2836.96, "text": "But back in the day, we're using it for the servlet. Remember at the beginning of the call when I was explaining how we exchange information to the RMI and the servlet, That was the part used for that." }, { "start": 2837.20001, "end": 2838.32, "text": "Okay. Gotcha." }, { "start": 2839.6, "end": 2849.84001, "text": "So that that's why it dropped. But you still need the part itself. And if you go at the very first page on the documentation, there is a very nice checklist of all the parts you need" }, { "start": 2852.225, "end": 2856.145, "text": "that we are I started. So if you go to docsactuary.com" }, { "start": 2856.145, "end": 2860.54501, "text": "prod MFT product, secure transport, there's a checklist with the ports." }, { "start": 2860.705, "end": 2864.56, "text": "It basically lists everything you need. So we don't drop" }, { "start": 2864.56, "end": 2867.84, "text": "any port per se. We actually need to add ports," }, { "start": 2868.0, "end": 2873.2, "text": "but we don't need the four four four or eight four four or whatever you're using for the admin UI" }, { "start": 2873.51999, "end": 2879.76, "text": "to be open between the nodes because the two admins don't talk to each other anymore. They just talk to to to their databases." }, { "start": 2880.375, "end": 2881.655, "text": "That's the big difference." }, { "start": 2882.295, "end": 2885.015, "text": "Okay. Gotcha. Thank you. Yep." }, { "start": 2885.735, "end": 2886.455, "text": "Okay." }, { "start": 2887.015, "end": 2899.08, "text": "The one port that is not in use anymore is the one down on if you are down on the server, there was another port called four four four three or something like that. That's the one we don't need anymore." }, { "start": 2899.64, "end": 2902.6, "text": "The TM communication, the two TMs talking to each other." }, { "start": 2902.92, "end": 2904.28, "text": "The RMI port," }, { "start": 2904.43999, "end": 2905.88, "text": "that's the one that drops," }, { "start": 2907.23999, "end": 2908.43999, "text": "but not the four four four." }, { "start": 2912.705, "end": 2913.66501, "text": "Makes sense?" }, { "start": 2915.58499, "end": 2917.745, "text": "Yes. Thank you. Okay." }, { "start": 2918.865, "end": 2921.185, "text": "I'm seeing a raised hand. Jean?" }, { "start": 2922.54, "end": 2923.74002, "text": "Good morning." }, { "start": 2924.38, "end": 2926.86002, "text": "Along that line, the" }, { "start": 2930.14, "end": 2931.02002, "text": "document" }, { "start": 2931.26, "end": 2935.66, "text": "that you just I've been going through the getting started guide because we're installing," }, { "start": 2936.06, "end": 2937.02002, "text": "net new" }, { "start": 2937.645, "end": 2941.565, "text": "Yep. To our dev servers now. What was the document that" }, { "start": 2941.565, "end": 2945.165, "text": "you just mentioned that lists all the ports? Because I've been trying to go through" }, { "start": 2945.72498, "end": 2946.76498, "text": "and and" }, { "start": 2947.00497, "end": 2948.365, "text": "get all my ports" }, { "start": 2948.76498, "end": 2958.84, "text": "so I can get my firewall guys to open them. Okay. And I've just run into yet another port that I did not have. Okay. So go to It's a little frustrating." }, { "start": 2958.91998, "end": 2959.56, "text": "Yeah." }, { "start": 2959.7200000000003, "end": 2963.0, "text": "Can you see my screen? So from the docs portal," }, { "start": 2963.24, "end": 2964.76, "text": "manage file transfer," }, { "start": 2965.0, "end": 2966.03998, "text": "secure transport." }, { "start": 2968.875, "end": 2970.315, "text": "Getting started." }, { "start": 2972.715, "end": 2973.51498, "text": "And" }, { "start": 2973.75497, "end": 2976.715, "text": "start working with the the initial configuration." }, { "start": 2977.435, "end": 2979.35498, "text": "And over here, there is a checklist." }, { "start": 2979.835, "end": 2980.1549999999997, "text": "Mhmm." }, { "start": 2981.08002, "end": 2983.24002, "text": "And over here are" }, { "start": 2983.56003, "end": 2991.80002, "text": "a gazillion and 17 ports for different things. Now Okay. Do you see seven the one we just ran into now is 7475." }, { "start": 2992.6, "end": 2995.24002, "text": "Oh, this one. Do we know about that one?" }, { "start": 2995.915, "end": 2997.11502, "text": "You" }, { "start": 2997.11502, "end": 2998.23502, "text": "do? I didn't." }, { "start": 2999.27502, "end": 3001.83502, "text": "ACN somewhere else at 7475." }, { "start": 3001.83502, "end": 3003.27502, "text": "Right? Yeah." }, { "start": 3004.23502, "end": 3005.59503, "text": "And it's in the" }, { "start": 3006.635, "end": 3008.635, "text": "it's in the admin guide I just found." }, { "start": 3009.3, "end": 3030.665, "text": "So I found a whole lot of ports, but is there a place that will list everything I need to begin with? Because it's certainly not wasn't fully in the getting started guide. Yeah. We have a workbook, and I don't know if it's it's not public, though. When when PSO are doing installs, we have our all the old workbook where we basically send it over." }, { "start": 3031.305, "end": 3042.7799999999997, "text": "I am not sure what happens when someone tries on their own. Let me take a note, I'll check, and I'll get back to you. So the answer is we know the information. I don't know if it's somewhere on a public place, so I just need to check." }, { "start": 3043.97998, "end": 3044.86, "text": "Oh, okay." }, { "start": 3045.65997, "end": 3052.53998, "text": "Thank you. I'll I'll let I'll let Byron know that too, by the way, so we can maybe get Yeah. So because this is frustrating." }, { "start": 3054.695, "end": 3066.65, "text": "Drop me you know, drop me a note if you don't hear from me in the next couple of days if I get distracted or if I if people don't respond to me and I forget to chase them. It's Yeah. I sure will. It's" }, { "start": 3067.05002, "end": 3074.09003, "text": "I know that we have the workbook because we were just reviewing it, and I'm pretty sure we have the list somewhere in a public place as well." }, { "start": 3074.57, "end": 3080.25, "text": "I just cannot it's probably somewhere on the support side. I just need to dig it out, and they don't want to spend half the call digging out." }, { "start": 3081.23502, "end": 3081.87503, "text": "And" }, { "start": 3082.11502, "end": 3085.39502, "text": "yeah. You know, work case scenario, I'll just send you the workbook anyway." }, { "start": 3086.035, "end": 3087.23502, "text": "Perfect. Thank you." }, { "start": 3087.71503, "end": 3093.315, "text": "Okay. But let me also verify internally and see what we want to do with that. That's the more important part." }, { "start": 3093.96002, "end": 3094.6, "text": "So" }, { "start": 3094.84, "end": 3101.48, "text": "but yes. And there is also a couple of high ports that are not listed there. They're in in the 9,000" }, { "start": 3101.48, "end": 3103.80002, "text": "range, nine six something or another." }, { "start": 3104.04, "end": 3106.12, "text": "You're you're still going to install with Oracle. Right?" }, { "start": 3106.815, "end": 3107.61502, "text": "Correct." }, { "start": 3108.0950000000003, "end": 3113.855, "text": "Yeah. There are a couple over there for the caches that we're using for the caching that are the range for the cache" }, { "start": 3114.575, "end": 3116.73502, "text": "that also get missed occasionally." }, { "start": 3117.295, "end": 3118.415, "text": "So" }, { "start": 3119.88, "end": 3125.56003, "text": "ping me. I'll I'll get you the complete set of ports, and then we'll figure out how to get that published somewhere." }, { "start": 3126.76, "end": 3129.80002, "text": "Perfect. I'll email you now. Mhmm." }, { "start": 3130.36002, "end": 3130.92, "text": "Okay." }, { "start": 3133.195, "end": 3135.515, "text": "Okay. What else do we have?" }, { "start": 3142.315, "end": 3143.115, "text": "No?" }, { "start": 3144.235, "end": 3149.035, "text": "Okay. Had anyone been playing with the new route packages" }, { "start": 3149.65, "end": 3150.77002, "text": "by any chance?" }, { "start": 3152.77, "end": 3153.81, "text": "Hey, Annie." }, { "start": 3154.37, "end": 3155.17, "text": "Hey." }, { "start": 3155.81, "end": 3162.05002, "text": "Oh, yeah. Not related to what you've literally just said, but I was just trying to find the unmute button at the time." }, { "start": 3163.57, "end": 3163.81, "text": "So" }, { "start": 3165.13498, "end": 3169.13498, "text": "we're seeing we're we're on a we're we're on 5.5," }, { "start": 3169.13498, "end": 3170.175, "text": "but we're on" }, { "start": 3170.815, "end": 3172.335, "text": "a much older patch level." }, { "start": 3172.495, "end": 3174.575, "text": "We're we're still in 2022," }, { "start": 3174.575, "end": 3177.855, "text": "so we're we're quite behind the times in that respect." }, { "start": 3178.99, "end": 3180.19, "text": "But we're seeing" }, { "start": 3180.67, "end": 3181.79, "text": "issues with" }, { "start": 3182.11, "end": 3185.1499599999997, "text": "when we hit the file tracking button" }, { "start": 3185.4700000000003, "end": 3187.4700000000003, "text": "to look at the file tracking log," }, { "start": 3188.91, "end": 3190.82996, "text": "it can it's taking sort of" }, { "start": 3191.475, "end": 3194.435, "text": "it can take sort of, like, twenty seconds or so," }, { "start": 3195.63495, "end": 3198.115, "text": "sometimes a bit more to actually come back" }, { "start": 3198.5149499999998, "end": 3202.2749599999997, "text": "with the results, but the server log most instant." }, { "start": 3202.355, "end": 3203.475, "text": "Yes." }, { "start": 3203.875, "end": 3207.07495, "text": "How so couple of questions. What kind of database do you have?" }, { "start": 3207.8999599999997, "end": 3209.5, "text": "It's an external" }, { "start": 3211.66, "end": 3212.62, "text": "SQL" }, { "start": 3212.77997, "end": 3213.58, "text": "database." }, { "start": 3214.1399499999998, "end": 3215.1, "text": "Okay. How" }, { "start": 3215.5, "end": 3218.2200000000003, "text": "many days do you keep in the tracking table?" }, { "start": 3219.18, "end": 3221.82, "text": "So in the file tracking, we got ninety days." }, { "start": 3222.695, "end": 3224.295, "text": "Ninety? Nine zero?" }, { "start": 3224.455, "end": 3225.895, "text": "Yeah. Nine zero." }, { "start": 3226.215, "end": 3228.695, "text": "How many files do you have per day?" }, { "start": 3230.855, "end": 3232.455, "text": "Not that many considering" }, { "start": 3232.855, "end": 3236.135, "text": "what some of these guys on here probably do." }, { "start": 3237.015, "end": 3238.135, "text": "I think" }, { "start": 3238.67, "end": 3239.4700000000003, "text": "we" }, { "start": 3241.87, "end": 3247.07, "text": "well, I do per month, it's about half a million, so it's not actually that many per day." }, { "start": 3247.79, "end": 3248.75, "text": "So Yeah." }, { "start": 3249.4700000000003, "end": 3252.91, "text": "It was. So your problem is that you're keeping too many days." }, { "start": 3253.855, "end": 3256.6549999999997, "text": "So the way our this" }, { "start": 3257.215, "end": 3262.09503, "text": "data bay the the file tracking database is not really designed to keep" }, { "start": 3262.415, "end": 3270.24, "text": "a lot of information in. And because of the way we're doing our indexing and so on based on the days with ninety days in," }, { "start": 3270.56, "end": 3277.2, "text": "it's that's part of your problem. So you have about one and a half million records, which is not that many." }, { "start": 3277.75995, "end": 3282.48, "text": "But even on that volume, I would start with lowering the days a little bit, honestly." }, { "start": 3284.245, "end": 3285.365, "text": "You can also" }, { "start": 3285.685, "end": 3288.725, "text": "see how much memory do you have in the admin UI." }, { "start": 3290.645, "end": 3294.885, "text": "Do you I I yeah. I don't know off the top of my head on that one." }, { "start": 3295.205, "end": 3304.0, "text": "I know it's in the configuration, but off top of my head, I'm not sure. Yeah. Get it up at least up to four gigabytes, if not higher, four to six." }, { "start": 3304.32, "end": 3312.0, "text": "See if that helps a little bit. But with that number of days kept in the database, the tracking table will always be slower than usual." }, { "start": 3312.815, "end": 3317.855, "text": "It's just we're we're not optimized. Our advice is no more than thirty days." }, { "start": 3318.415, "end": 3322.735, "text": "Forty five for very small environments, your yours will qualify in a way." }, { "start": 3323.535, "end": 3331.04004, "text": "I would so it's not just the number of records. So because based on the number of records, I'm okay with you keeping almost 90." }, { "start": 3331.2, "end": 3334.72003, "text": "Right? With the 1 and a half mil, that's not a problem usually," }, { "start": 3335.36, "end": 3337.6000400000003, "text": "but it's just not optimized" }, { "start": 3337.6000400000003, "end": 3342.56, "text": "for going to that many days and months because of how it's done behind the scenes. And" }, { "start": 3343.255, "end": 3353.255, "text": "the other thing is and you'll know I'll say that, update your server. We had been doing a lot of work on this table optimizing it. So the newer version should work a little faster." }, { "start": 3353.575, "end": 3357.415, "text": "But still, I I strongly recommend not to keep ninety days." }, { "start": 3359.19, "end": 3363.75, "text": "Alright. That that's that's really helpful. Thank you, Annie. So one of the reasons" }, { "start": 3364.70996, "end": 3367.26996, "text": "we felt the need to have" }, { "start": 3367.43, "end": 3369.67, "text": "that number of days is because we sometimes" }, { "start": 3370.385, "end": 3374.865, "text": "have our external users coming in and asking us questions, which" }, { "start": 3375.665, "end": 3379.825, "text": "can sometimes mean that we need to look at the data" }, { "start": 3380.145, "end": 3381.985, "text": "older than thirty days old" }, { "start": 3383.0249599999997, "end": 3384.945, "text": "to do some analysis work and stuff." }, { "start": 3386.44, "end": 3391.24005, "text": "Well So one of the things that we are looking at doing is is the possibility of" }, { "start": 3391.56006, "end": 3393.24005, "text": "sending our logs to" }, { "start": 3393.56006, "end": 3394.44006, "text": "Splunk" }, { "start": 3394.52, "end": 3403.2649499999998, "text": "and do some analysis there rather than just keeping it in the database. Would is that, like, a sort of a thing that you'd recommend rather than keeping it all in the database?" }, { "start": 3403.90497, "end": 3413.58496, "text": "So the tracking table is not really locks. It's data objects. So sending the tracking data to Splunk will not be automatic. You can send the server locks to Splunk without trouble." }, { "start": 3414.26, "end": 3418.66003, "text": "But if you want to send the tracking table data, you'll need to first read it somehow." }, { "start": 3419.54004, "end": 3421.46, "text": "One of the ways through the APIs." }, { "start": 3421.94, "end": 3424.5, "text": "Another way is to look at" }, { "start": 3424.74, "end": 3425.54004, "text": "Sentinel," }, { "start": 3425.54004, "end": 3428.26, "text": "our other server, the one that is basically can" }, { "start": 3428.585, "end": 3431.86505, "text": "keep tracking records for as long as you want." }, { "start": 3432.10504, "end": 3433.86505, "text": "It's basically designed for that." }, { "start": 3434.34503, "end": 3440.825, "text": "And we have the new event router that now can send to Kafka, and they are also looking at other options where it can send." }, { "start": 3441.145, "end": 3443.32, "text": "So that maybe downstream," }, { "start": 3443.32, "end": 3444.76, "text": "that will be the next option." }, { "start": 3445.56, "end": 3455.88, "text": "But, yes, short term, if you create something that consumes the tracking table and copies it to Splunk or sends it to Splunk, that might be the cleanest way to do it for now." }, { "start": 3456.855, "end": 3462.615, "text": "Is it is it possible to because I know through the Log four j configuration," }, { "start": 3463.415, "end": 3465.975, "text": "we can write Yeah. The file." }, { "start": 3466.695, "end": 3469.895, "text": "Can we is it possible to write the file tracking to" }, { "start": 3470.13, "end": 3473.73004, "text": "a file and then just one can test that? Nope." }, { "start": 3473.89, "end": 3477.01, "text": "What kind of information do they usually look for it for?" }, { "start": 3478.5299999999997, "end": 3479.33, "text": "It's" }, { "start": 3480.13, "end": 3481.73, "text": "when they're looking for" }, { "start": 3484.46497, "end": 3487.9049999999997, "text": "issues that they may have had submitting files to us." }, { "start": 3488.865, "end": 3493.3450000000003, "text": "And sometimes they're very delayed in asking these questions," }, { "start": 3493.425, "end": 3499.88, "text": "which is frustrating for us, but Yeah. We have to deal with it because they're we kind of, like, supply a service to them. So" }, { "start": 3500.68, "end": 3501.4, "text": "Okay." }, { "start": 3501.56, "end": 3505.4, "text": "Well, as I said, you might want to look to the transfers API" }, { "start": 3505.56, "end": 3509.08, "text": "that will allow you to export the tracking table in a JSON format," }, { "start": 3510.755, "end": 3516.755, "text": "or you can just export the tracking table in an Excel format directly from the UI, especially with your number of records." }, { "start": 3516.91504, "end": 3522.35504, "text": "And then this information can be saved somewhere, or you can pass it and send it somewhere daily, for example." }, { "start": 3523.02, "end": 3526.54004, "text": "So, for example, keep the last thirty days in the database," }, { "start": 3526.94, "end": 3527.74, "text": "and then on" }, { "start": 3528.38, "end": 3531.1000400000003, "text": "for the last five days, just pass the information" }, { "start": 3531.42004, "end": 3537.34, "text": "and send it to Splunk or another database or somewhere out of ST, out of the operational days ST." }, { "start": 3537.945, "end": 3540.10504, "text": "That might be your cleanest way maybe." }, { "start": 3540.665, "end": 3543.2250400000003, "text": "Okay. So log four j is not an option?" }, { "start": 3543.465, "end": 3548.9049999999997, "text": "No. It's not a log four j lock. That's the problem. It's not a log file at all. We call it a transfer log," }, { "start": 3549.145, "end": 3550.34503, "text": "but that's deceiving." }, { "start": 3552.11, "end": 3563.39, "text": "Right. Yes. That's probably that's probably where our confusion is then. Oh, that's lovely. Alright. Thank you, Annie. Yeah. And if so are you on prem or are you in oh, you're on prem with this version?" }, { "start": 3564.065, "end": 3565.3450000000003, "text": "Yeah. We are. Yeah." }, { "start": 3566.065, "end": 3568.785, "text": "Do you have entitlement for Sentinel by any chance?" }, { "start": 3571.58496, "end": 3573.985, "text": "It's it's something that I think we've" }, { "start": 3574.46497, "end": 3576.305, "text": "kind of looked at before." }, { "start": 3576.945, "end": 3581.59, "text": "I I'd I'd love to expand our Axewate products, but I'm" }, { "start": 3581.59, "end": 3582.87006, "text": "sort of limited" }, { "start": 3583.43005, "end": 3587.83, "text": "with budget and stuff like that. So I'm not sure that would be an option." }, { "start": 3588.47003, "end": 3594.79004, "text": "Well, maybe not short term, but I would say look at it again if you want to because it's exactly for this kind of scenarios" }, { "start": 3595.055, "end": 3601.935, "text": "because it can keep it it's essentially a huge database that you can send all the tracking information. You need goals in real time," }, { "start": 3602.65497, "end": 3606.815, "text": "and you can get all the information even years later if need be." }, { "start": 3607.83, "end": 3608.71, "text": "So that" }, { "start": 3609.75, "end": 3625.555, "text": "and I'm don't get me wrong. I I'm not saying that is the only option. I'm just mentioning that that's part of the reason why we will never redesign tracking table to hold that much information is because our policy is for the longer term, GoSentinel or something else," }, { "start": 3625.795, "end": 3633.875, "text": "go out. But we're also working on a couple of connectors to the event router so that we can actually push to other engines, not just ours." }, { "start": 3633.955, "end": 3635.475, "text": "But we are not not there yet." }, { "start": 3637.35, "end": 3641.83, "text": "Hey, Andy. Are are you Yeah. Are people still writing out to the expert log?" }, { "start": 3642.47003, "end": 3648.87, "text": "We are. That's why I asked what kind of information he needs. The problem with expert log is that it's not connected." }, { "start": 3649.0299999999997, "end": 3652.15, "text": "So they you can see the inbound and you can see the outbound," }, { "start": 3652.875, "end": 3655.91504, "text": "but there is nothing to tell you what belongs to what." }, { "start": 3657.195, "end": 3663.91504, "text": "Yeah. So my No. But we we use that in Splunk a lot. And and so if you're just looking to see if you're" }, { "start": 3665.03503, "end": 3666.715, "text": "for folks that call" }, { "start": 3667.115, "end": 3667.835, "text": "us after," }, { "start": 3668.37, "end": 3670.45, "text": "you know, our forty five days," }, { "start": 3671.49, "end": 3674.93, "text": "we have to go look in Splunk for their file names," }, { "start": 3675.41, "end": 3680.29, "text": "and we'll be able to see if it was delivered or not. Yeah. But you're right. I won't get any." }, { "start": 3681.325, "end": 3692.205, "text": "Well, there was a p the PGP decryption error or, you know, something like that. No. But if it is just about did I got the file or not, Jean is right. So, Machu, if you look on the OS level," }, { "start": 3692.525, "end": 3696.685, "text": "there is another file called expert lock. It lives on all servers and all edges." }, { "start": 3698.42, "end": 3705.0599, "text": "Yeah. I I and I found I found that because I thought that might actually be our answer Yeah. A while ago." }, { "start": 3705.54, "end": 3712.065, "text": "But looking in in that log, it didn't seem to show everything that we might actually be looking for." }, { "start": 3712.3050000000003, "end": 3715.9849999999997, "text": "I I get the I get the point of it being there and and" }, { "start": 3716.465, "end": 3719.585, "text": "some of the data that's held in it, but it didn't seem to" }, { "start": 3719.9049999999997, "end": 3723.3450000000003, "text": "have everything that the track the file tracking" }, { "start": 3723.425, "end": 3743.755, "text": "screen would show. No. It doesn't. It doesn't. It it what it does is basically just transfer itself. So it tell you a file called Annie arrived from this IP on this date on this protocol, but it doesn't tell you if Annie originated the transfer or if the file was pulled from Annie. So it's just a pure transfer log." }, { "start": 3743.995, "end": 3753.035, "text": "So but and Jean is bringing it up, and I was getting there as well at some point maybe, or I I discarded it, but Jean is right why I should have brought it up." }, { "start": 3753.8, "end": 3764.0398999999998, "text": "It actually is easy enough to send to Splunk, especially because it's a rotating file. So it rotates overnight, so you can send yesterday's logs cleanly because they're outside of even" }, { "start": 3765.48, "end": 3767.08, "text": "if it doesn't give you everything," }, { "start": 3767.6549999999997, "end": 3769.7349999999997, "text": "at least it gives you the," }, { "start": 3769.9751, "end": 3778.295, "text": "hey. How many files did this user sent on this date? Things like that will actually be there. It will not tell you what happened to the files for the most part," }, { "start": 3778.535, "end": 3782.6951, "text": "but at least you can get the basic curious, did you get my file on Tuesday?" }, { "start": 3783.12, "end": 3784.24, "text": "Things like that." }, { "start": 3784.6400000000003, "end": 3790.48, "text": "Mhmm. Yeah. I I think you're right. I think it's something that we might need to take another look at and" }, { "start": 3791.2799999999997, "end": 3791.9199, "text": "and" }, { "start": 3792.3199999999997, "end": 3795.0398999999998, "text": "and sort of compare that with the kind of" }, { "start": 3795.3599999999997, "end": 3797.6800000000003, "text": "requests and questions that we get from our external" }, { "start": 3798.625, "end": 3804.6249, "text": "if we can answer them from the stuff that's in there. But yeah. No. Thank you, and thank you, June, for your answers." }, { "start": 3805.025, "end": 3812.9449999999997, "text": "And it's not all or nothing. You know? You can always because you you're a service, you can always basically tell him if you're forty days or later," }, { "start": 3813.4601000000002, "end": 3817.2200000000003, "text": "all you can tell you is if we got the file or at least temporary." }, { "start": 3817.38, "end": 3820.66, "text": "And then if you are exporting your server logs" }, { "start": 3821.06, "end": 3822.98, "text": "or if they're on file system," }, { "start": 3823.38, "end": 3838.8049, "text": "once you find the the way I usually work in these cases, if I need to do something like that, I'll hit the extra lock first to see if the file is there and at what time because that gives you a timeline of when to look into the server lock export or server lock files." }, { "start": 3839.125, "end": 3839.4451, "text": "Right?" }, { "start": 3841.27, "end": 3849.7499, "text": "Yeah. Those types names are important. And especially with the newer versions when you get to that civilized versions of ST a k 2024," }, { "start": 3849.99, "end": 3851.19, "text": "I have to say that." }, { "start": 3851.59, "end": 3852.1499, "text": "But" }, { "start": 3852.4700000000003, "end": 3867.3549000000003, "text": "we updated our Xferlock rights. So you can act we actually built a second expert lock as well for the team specifically with a little bit more details, including threat IDs and stuff like that to make it easier to match into" }, { "start": 3867.6749, "end": 3868.475, "text": "server locks." }, { "start": 3869.9099, "end": 3875.9099, "text": "So once you update to a newer server, you can actually weaponize your extra log a lot better." }, { "start": 3876.0699999999997, "end": 3880.0699999999997, "text": "The old one is still there, but we also added more fields for the TM driven one." }, { "start": 3882.635, "end": 3886.5549, "text": "Okay. That's interesting. Yeah. Look out for that on the on the later versions." }, { "start": 3887.0349, "end": 3894.7949, "text": "Yeah. Which will make it easier then to match to server locks that you also probably are sending somewhere or at least you are hiding somewhere." }, { "start": 3895.115, "end": 3895.835, "text": "So it's" }, { "start": 3896.16, "end": 3900.0, "text": "looking for all the information will always be a problem no matter what." }, { "start": 3900.3201, "end": 3900.88, "text": "But" }, { "start": 3901.3599999999997, "end": 3906.96, "text": "and, again, if if all the problem you have is that the database is just a little bit slow," }, { "start": 3907.3599999999997, "end": 3908.0801, "text": "though so" }, { "start": 3908.775, "end": 3912.615, "text": "that's another thing I was going to ask you. When you are looking for this" }, { "start": 3912.8550999999998, "end": 3918.855, "text": "anything, is it always slow, or is it only for if you're looking for a specific date range or specific file name?" }, { "start": 3919.7349999999997, "end": 3921.495, "text": "It seems to be always." }, { "start": 3922.1800000000003, "end": 3927.5398999999998, "text": "And no matter what kind of filtering we put on it Mhmm. Every time we refresh," }, { "start": 3927.62, "end": 3935.7799999999997, "text": "it it it seems to take the same amount of time. It's quite it is, you know, to to ten, twenty, twenty five seconds, something like that to come back." }, { "start": 3936.755, "end": 3953.1599, "text": "So as I said, first of all, get some more memory into it. Another test to do is run it with a query that covers a single day. So run something with give me file give me all the files that are in the last hour or that are something like that, always that is inside of the same day" }, { "start": 3954.0398999999998, "end": 3956.6799, "text": "because we're splitting the logs per day." }, { "start": 3957.0, "end": 3960.1198999999997, "text": "So whenever we are crossing days is when everything will get slow." }, { "start": 3960.44, "end": 3964.6799, "text": "And the generic query that doesn't check that just says, give me everything. You know?" }, { "start": 3965.9249, "end": 3967.1249, "text": "But that's one thing." }, { "start": 3967.4449999999997, "end": 3976.325, "text": "But as I said, ninety days is usually a bit too much. It it you're not over on the records because that's about when I tell people to start thinking team." }, { "start": 3976.645, "end": 3979.2849, "text": "So if all you have is 1,500,000," }, { "start": 3979.4449999999997, "end": 3981.7799, "text": "you should be general yes" }, { "start": 3981.94, "end": 3982.74, "text": "and no." }, { "start": 3983.1400000000003, "end": 3985.8599999999997, "text": "Because the if if you have 1,500,000" }, { "start": 3985.8599999999997, "end": 3993.0599, "text": "records, you probably have about double that lines. So 3,000,000 lines is only it's about about twice as I like to see in this tracking table." }, { "start": 3994.4049999999997, "end": 3994.725, "text": "So" }, { "start": 3995.7650000000003, "end": 3997.9249, "text": "Cool. That's lovely. Thanks, Annie." }, { "start": 3998.245, "end": 3999.0449, "text": "Okay." }, { "start": 3999.285, "end": 4005.0449, "text": "Good luck figuring it out. And, again, update at some point because there were some improvements that will help you," }, { "start": 4005.605, "end": 4008.325, "text": "but also explore outside of the tracking table." }, { "start": 4009.5701, "end": 4016.9300000000003, "text": "Yeah. That that that was gonna be, like, another question if but I I wanted other people to have a chance to answer a question before I ask" }, { "start": 4016.9300000000003, "end": 4020.21, "text": "that one. It was just about the the path to update" }, { "start": 4020.5299999999997, "end": 4023.3301, "text": "from our version and and what we should expect and stuff." }, { "start": 4024.005, "end": 4024.885, "text": "Okay." }, { "start": 4024.885, "end": 4029.605, "text": "So the short version is the officially, we test up to twelve months." }, { "start": 4030.245, "end": 4037.7650000000003, "text": "So you cannot jump more than twelve months on a single update, so you need to do updates, intermediate update to get to where we are." }, { "start": 4039.5699999999997, "end": 4040.45, "text": "However," }, { "start": 4040.45, "end": 4043.6499, "text": "support had been testing some other parts occasionally." }, { "start": 4044.13, "end": 4052.13, "text": "So with as old as you are as your releases, I would strongly recommend just open a ticket to support and ask them for a recommended part." }, { "start": 4054.4849999999997, "end": 4058.9651, "text": "Yeah. That's I think I think that's probably the the best way to go for us at the moment. So" }, { "start": 4059.6050999999998, "end": 4060.885, "text": "Cool. Thanks a lot." }, { "start": 4061.2051, "end": 4071.8599, "text": "Yeah. Because if they some they have r and d behind the scenes, so r and d might have a better idea. They know which builds to skip here and there. Right?" }, { "start": 4072.1, "end": 4075.3799, "text": "But be prepared for multiple update." }, { "start": 4075.62, "end": 4093.815, "text": "The good news is that that updates are fast. The bad news is that you got you said you have enterprise cluster. Right? Yeah. But you have edges. So don't forget that September, you will update the edges to the new database as well. So the September release on the edges the September update on the edges or later will be a little slower." }, { "start": 4095.0950000000003, "end": 4097.495, "text": "So you need to September 20 right?" }, { "start": 4099.02, "end": 4100.5, "text": "That's September" }, { "start": 4100.5, "end": 4101.5801, "text": "24." }, { "start": 4101.5801, "end": 4102.14, "text": "Yes." }, { "start": 4102.62, "end": 4103.4199, "text": "Okay." }, { "start": 4105.42, "end": 4107.1, "text": "Okay. Cool. Thanks, Nay." }, { "start": 4107.5801, "end": 4108.2999, "text": "Absolutely." }, { "start": 4108.78, "end": 4110.22, "text": "Okay. Idi," }, { "start": 4110.62, "end": 4114.78, "text": "TIM graceful shutdown. Yay. My favorite topic. We fixed that." }, { "start": 4116.085, "end": 4121.3649000000005, "text": "So so Edi is asking, can we talk about the transaction manager Graceful shutdown and what happens?" }, { "start": 4121.6849, "end": 4129.2999, "text": "So Graceful shutdown was built five years ago or more as our first attempt at zero downtime," }, { "start": 4129.78, "end": 4131.54, "text": "it was done very stupidly." }, { "start": 4131.54, "end": 4132.98, "text": "It was essentially" }, { "start": 4134.74, "end": 4143.215, "text": "tell TM that it is going to shut down in thirty minutes, so we stop receiving new connections or new jobs. And then exactly thirty minutes later," }, { "start": 4143.9349, "end": 4147.775, "text": "even if something is still running, we'll shut down, but we can shut down earlier" }, { "start": 4147.775, "end": 4154.255, "text": "if something is still going, which was a pain in the behind because if you had a three hours long transfer, it was cut in the middle." }, { "start": 4155.02, "end": 4158.8598999999995, "text": "So when we started working on ZGO this year or last year," }, { "start": 4159.1, "end": 4165.7399000000005, "text": "they realized that, yeah, we need to fix that. So the new graceful shutdown is actually intelligent." }, { "start": 4166.38, "end": 4173.0650000000005, "text": "When you start the Graceful shutdown, the transaction manager stops receiving new jobs from its protocols" }, { "start": 4173.545, "end": 4176.1849, "text": "and stops picking jobs from the database." }, { "start": 4177.385, "end": 4178.1849, "text": "However," }, { "start": 4178.4249, "end": 4182.505, "text": "any job already in the transaction manager is going to be completed" }, { "start": 4182.8201, "end": 4184.34, "text": "before we shut down." }, { "start": 4184.74, "end": 4191.9400000000005, "text": "So if you have a push or a pull going on that might take three more days to finish because you're running a terabyte file from somewhere" }, { "start": 4192.26, "end": 4193.0599, "text": "weird," }, { "start": 4193.86, "end": 4199.5149, "text": "it will still this TM will keep jogging and working only on this specific transfer" }, { "start": 4199.675, "end": 4201.755, "text": "until it finishes completely." }, { "start": 4203.515, "end": 4208.955, "text": "If it fails for some reason, it will not restart. It will put it in the database and it will consider" }, { "start": 4208.955, "end": 4209.755, "text": "it" }, { "start": 4210.14, "end": 4211.02, "text": "finished." }, { "start": 4211.74, "end": 4219.8999, "text": "If it finishes, the processing itself will be thrown to the database, and it will be sent forever. So it's just the current operation that needs to finish." }, { "start": 4220.38, "end": 4226.8150000000005, "text": "If we're in the middle of a advanced routing that has PGP followed by ZIP followed by push or whatever," }, { "start": 4226.975, "end": 4229.7749, "text": "it will do the whole advanced routing part" }, { "start": 4230.415, "end": 4235.455, "text": "and then finish the process so it it can close the advanced routing." }, { "start": 4236.095, "end": 4238.735, "text": "As we know, when you go into deliveries," }, { "start": 4240.07, "end": 4240.63, "text": "if," }, { "start": 4241.4299, "end": 4249.35, "text": "depending on the the scenario, that might mean waiting them out as well, including all of the retries. If the retries are inside the far route," }, { "start": 4249.91, "end": 4261.1649, "text": "obviously, we'll need to wait them out. So this retries will run. But the idea of the graceful shutdown is that it allows the server to finish what it had started without interruptions." }, { "start": 4261.725, "end": 4264.7649, "text": "And this means if a file is coming from an end user" }, { "start": 4265.085, "end": 4268.0449, "text": "and they're uploading or downloading, we're not going to cut it." }, { "start": 4269.24, "end": 4270.4400000000005, "text": "Excellent, CD?" }, { "start": 4271.0, "end": 4271.6399, "text": "Yes." }, { "start": 4272.28, "end": 4274.36, "text": "It's A couple follow ups." }, { "start": 4274.6, "end": 4277.24, "text": "Mhmm. So in the event that," }, { "start": 4277.8, "end": 4280.04, "text": "say, a folder monitor" }, { "start": 4280.04, "end": 4281.88, "text": "is is hung" }, { "start": 4283.1051, "end": 4284.9451, "text": "and you trigger" }, { "start": 4284.9451, "end": 4285.665, "text": "the" }, { "start": 4286.0651, "end": 4287.505, "text": "graceful shutdown," }, { "start": 4287.665, "end": 4289.025, "text": "will that" }, { "start": 4289.425, "end": 4290.225, "text": "hung" }, { "start": 4290.385, "end": 4291.345, "text": "event" }, { "start": 4291.425, "end": 4292.385, "text": "prevent" }, { "start": 4292.7051, "end": 4296.625, "text": "the TM from fully shutting down, or is there, like, a time out" }, { "start": 4299.5, "end": 4302.6199, "text": "time clock on those? It shouldn't," }, { "start": 4302.7, "end": 4305.2599, "text": "but I can guarantee you no one tested that." }, { "start": 4305.659900000001, "end": 4306.3799, "text": "Okay." }, { "start": 4308.14, "end": 4316.335, "text": "I'll be honest here. Right? Because getting it to hang is not that easy, so I'm pretty sure no one tested that. In theory," }, { "start": 4316.895, "end": 4319.7749, "text": "hunk objects should be cleaned up sooner or later." }, { "start": 4320.015, "end": 4336.04, "text": "And because the team is not picking up more jobs, it should be timing out all the threats at this point. And because it is, you know, shut down, it should have passed that folder monitor and schedule work control. If it is the first server, the one that was holding the folder monitor and schedule work services," }, { "start": 4336.36, "end": 4338.68, "text": "they should have during the shutdown," }, { "start": 4339.335, "end": 4344.8549, "text": "the moment when this is issued, the those two services should have flown to another server in the cluster anyway." }, { "start": 4345.1749, "end": 4346.135, "text": "So that should" }, { "start": 4346.775, "end": 4350.695, "text": "that, in theory, should be enough to actually resolve the deadlock." }, { "start": 4352.4501, "end": 4356.21, "text": "Okay. And if that scenario does exist, it actually should" }, { "start": 4356.4501, "end": 4358.13, "text": "show up in the" }, { "start": 4360.2101, "end": 4361.25, "text": "event queue," }, { "start": 4361.41, "end": 4365.3301, "text": "and we should be able to to kill it in" }, { "start": 4366.1951, "end": 4369.955, "text": "in in that portion. Is that am I thinking correctly?" }, { "start": 4370.515, "end": 4374.595, "text": "So if it is in the event queue, it's not being processed at the moment." }, { "start": 4375.0751, "end": 4379.01, "text": "So the event queue has the waiting list. What is waiting?" }, { "start": 4379.3301, "end": 4384.37, "text": "So the moment when you do the graceful shutdown, you're telling you're telling this TM," }, { "start": 4384.4501, "end": 4397.785, "text": "stop picking up from the event queue. So it can still write to the event queue. So if something happens, for example, it gets the file that needs processing, it will put the incoming end into the event queue, but it will not pick it up again because it knows not to." }, { "start": 4398.665, "end": 4402.585, "text": "But if an event is already taken by a database," }, { "start": 4402.9851, "end": 4406.905, "text": "technically, you will not see it in in the event queue. It's in the memories." }, { "start": 4408.91, "end": 4414.909900000001, "text": "Okay. And there are a couple of exceptions, of course. But for the case we are talking about, that's basically" }, { "start": 4415.15, "end": 4421.3899, "text": "it's it's it's the the hanging is not really an event event. It's basically the the" }, { "start": 4421.55, "end": 4423.15, "text": "folder monitor service itself." }, { "start": 4424.325, "end": 4424.965, "text": "And" }, { "start": 4425.365, "end": 4430.5650000000005, "text": "my guess, and, again, haven't been tested, and I don't know how to test it to even confirm." }, { "start": 4430.8049, "end": 4438.245, "text": "But I would expect that the pure process of switching the services into the other node will be enough to resolve the problem." }, { "start": 4438.85, "end": 4443.1699, "text": "Because for all intents and purposes, it looks like this TM just shut down." }, { "start": 4444.6901, "end": 4445.8099, "text": "Oh, okay." }, { "start": 4446.53, "end": 4450.77, "text": "Okay. We had a a we we had" }, { "start": 4452.245, "end": 4454.8049, "text": "an issue that came up when we" }, { "start": 4455.445, "end": 4456.325, "text": "applied" }, { "start": 4456.4048999999995, "end": 4459.6849, "text": "our our the July patch to our production. And" }, { "start": 4461.365, "end": 4463.365, "text": "there were things in the logs," }, { "start": 4464.23, "end": 4467.4299, "text": "errors in the or no. Actually, there were warns in the logs" }, { "start": 4468.15, "end": 4477.51, "text": "that we did not see in our stage environment when we upgrade. Although our stage environment is not not even close to being as busy as our production." }, { "start": 4478.5751, "end": 4479.215, "text": "And" }, { "start": 4479.855, "end": 4485.775, "text": "some of some of them, there were, like, nine events that talked about a heartbeat" }, { "start": 4486.255, "end": 4487.375, "text": "is missing," }, { "start": 4488.3351, "end": 4489.535, "text": "and that kept" }, { "start": 4489.535, "end": 4492.895, "text": "writing in the TM log, like, every ten seconds." }, { "start": 4494.2001, "end": 4495.7999, "text": "And then there was also" }, { "start": 4498.28, "end": 4499.7999, "text": "a a a poll." }, { "start": 4499.9601, "end": 4501.88, "text": "A poll tried to initiate," }, { "start": 4501.9601, "end": 4505.96, "text": "but it said that it was still in process, and we'll wait for the next schedule." }, { "start": 4506.52, "end": 4511.345, "text": "But that never did clear up. So that just kept writing." }, { "start": 4512.0650000000005, "end": 4516.3049, "text": "I think we went in there manually and fixed a couple of them." }, { "start": 4516.705, "end": 4517.5049, "text": "And so" }, { "start": 4518.385, "end": 4519.825, "text": "I was thinking that" }, { "start": 4520.145, "end": 4522.225, "text": "if we use the" }, { "start": 4522.225, "end": 4523.4249, "text": "graceful shutdown," }, { "start": 4523.98, "end": 4526.4598000000005, "text": "those events that were still pending" }, { "start": 4526.86, "end": 4533.74, "text": "or got caught in in that would resolve that. Do you think that that assessment" }, { "start": 4533.74, "end": 4534.6199, "text": "is true?" }, { "start": 4535.5, "end": 4551.005, "text": "Don't know. Okay. I I because just because I don't know the root cause. So the ones that were telling you that the pool is waiting for the next attempt, this one will be the event queue. So for this one, you could have found it and killed it from the event queue." }, { "start": 4551.8099, "end": 4552.45, "text": "Okay." }, { "start": 4552.7699, "end": 4556.6098999999995, "text": "Okay. So that being the case, so if I do a graceful shutdown" }, { "start": 4556.7699, "end": 4558.1299, "text": "on one node," }, { "start": 4559.33, "end": 4565.25, "text": "I should be able to look in the event queue. And if there's anything in there, will I be able to" }, { "start": 4566.215, "end": 4569.095, "text": "delete them having the TM" }, { "start": 4569.415, "end": 4570.2949, "text": "service" }, { "start": 4570.775, "end": 4571.735, "text": "offline?" }, { "start": 4574.135, "end": 4578.0549, "text": "The TM service offline, yes, because you still have the admin service." }, { "start": 4578.135, "end": 4582.6199, "text": "But don't don't forget that the queue is shared between the nodes." }, { "start": 4582.86, "end": 4587.659900000001, "text": "So as long as the other TM is still running, there will be events in that queue," }, { "start": 4588.1399, "end": 4589.9799, "text": "and you don't want to kill those." }, { "start": 4590.7, "end": 4591.4199, "text": "Okay." }, { "start": 4591.659900000001, "end": 4598.945, "text": "Okay. So kind of our oh, go ahead. No. No. Go I was going to say, if you think that you don't have anything pending," }, { "start": 4598.9451, "end": 4602.865, "text": "so you have you don't have any files that are waiting to be pushed and so on," }, { "start": 4603.5851, "end": 4606.145, "text": "if you shut down all the transaction managers," }, { "start": 4606.7201000000005, "end": 4614.48, "text": "the event queue should only contain kings that are supposed to be there. For example, file arrived, but didn't get processed or retries waiting." }, { "start": 4615.2001, "end": 4619.1201, "text": "So if you so anything that you don't expect can be cleaned." }, { "start": 4619.735, "end": 4624.9349, "text": "So you don't need the running TM to clean the event queue because the event queue itself" }, { "start": 4625.735, "end": 4630.455, "text": "is in the database, and the admin UI actually goes directly into the database for it." }, { "start": 4630.775, "end": 4635.71, "text": "The TM is the reader and the writer, but the queue itself is a database table." }, { "start": 4636.35, "end": 4637.07, "text": "Okay." }, { "start": 4637.15, "end": 4640.4299, "text": "So you don't need to change it. So it sounds like" }, { "start": 4640.67, "end": 4642.5099, "text": "when I when I patch," }, { "start": 4643.07, "end": 4645.15, "text": "I should gracefully shut down" }, { "start": 4646.4349999999995, "end": 4647.5549, "text": "the TM" }, { "start": 4647.875, "end": 4653.875, "text": "on one node. Once that's completed, shut down the TM great the graceful shutdown on" }, { "start": 4653.875, "end": 4655.235, "text": "the second node," }, { "start": 4655.715, "end": 4660.275, "text": "and then look in the in the event queue if there's anything in there, delete those things," }, { "start": 4660.95, "end": 4664.7899, "text": "and then go to the server itself and stop" }, { "start": 4665.27, "end": 4665.9899000000005, "text": "all." }, { "start": 4666.3099, "end": 4685.965, "text": "Yeah. Again, Edi, if you do that and the file the if the last operation that happens is a file arriving, the processing will not happen because you'll delete it. So I wouldn't do it that way. I understand where you're coming from, and I know that something got really weird. The idea of the graceful shutdown, it was built specifically for the zero downtime update" }, { "start": 4686.125, "end": 4697.9, "text": "so that we can do the rolling update. Right? In your case, what I would do is just shut down one of the t the TMs together if you're not going to going to do a ZDO," }, { "start": 4698.3, "end": 4710.995, "text": "and just do the regular update. It what you saw in July shouldn't happen. Let's say it like that. And if you delete so if you decide to do it with the Grateful shutdown, don't forget that when the file arrives" }, { "start": 4711.3150000000005, "end": 4712.8351999999995, "text": "with the Grateful shutdown," }, { "start": 4712.915, "end": 4728.4302, "text": "we will throw an event into the database that says this file need processing and will shut down because we finished the arrival part. Right? Mhmm. Mhmm. If you delete it, this file will never get processed. So it will remain in progress in your tracking table forever," }, { "start": 4728.9102, "end": 4734.030000000001, "text": "and it will just sit on the file system. And there is nothing to be done for this file except someone to reupload it." }, { "start": 4735.355, "end": 4747.6748, "text": "That's why I don't want to go and say go delete everything in the tracking, because the event queue is supposed to have events at all times. This is the list of things that still need to happen. If you have a long retry" }, { "start": 4748.71, "end": 4752.3901000000005, "text": "that is on the waiting list, it will also disappear. So things like that." }, { "start": 4752.79, "end": 4753.9102, "text": "Uh-huh. But" }, { "start": 4754.469999999999, "end": 4766.705, "text": "once it starts, if if you have a subscription that is complaining that it's waiting for the next attempt, this is what you need to get the ID and go find it in the tracking in the event queue, and then you can queue it from there manually." }, { "start": 4767.7449, "end": 4768.385, "text": "Okay." }, { "start": 4769.025, "end": 4774.705, "text": "More of a, you know, surgical strike than just take it out everything just because," }, { "start": 4774.945, "end": 4775.745, "text": "otherwise," }, { "start": 4776.145, "end": 4777.105, "text": "things go weird." }, { "start": 4778.280000000001, "end": 4781.0, "text": "Okay. Now if you have the time," }, { "start": 4781.8801, "end": 4787.32, "text": "what I like doing during updates is if you if you're gonna take an outage anyway," }, { "start": 4787.719999999999, "end": 4793.0, "text": "shut down all of the so I would shut down the scheduler and" }, { "start": 4794.195, "end": 4797.1553, "text": "inbound ports about five minutes before I start." }, { "start": 4797.3152, "end": 4799.715, "text": "This will kill all new files coming in," }, { "start": 4801.0752, "end": 4818.21, "text": "and you can also kill the folder monitor at this point so that you leave the TMs to just finish whatever files are in the system at the moment. And then you can monitor the event queue to see what you see is still there before the final stop. So if you're worried about stuck jobs, that's what I would do more than going with the Graysville shutdown for now." }, { "start": 4819.25, "end": 4820.530000000001, "text": "Or that's a great" }, { "start": 4820.8499, "end": 4824.5051, "text": "that is a great strategy. I'm gonna adopt that into my process." }, { "start": 4824.905000000001, "end": 4825.785, "text": "Just kill" }, { "start": 4826.344999999999, "end": 4836.1849999999995, "text": "if you have an hour time, you might as well, while you're preparing, just kill everyone the first ten minutes so that ST can still work these ten minutes, finish whatever it had started," }, { "start": 4836.344999999999, "end": 4838.5051, "text": "but kill all inbound files earlier." }, { "start": 4839.43, "end": 4840.87, "text": "That that's cleaner." }, { "start": 4840.95, "end": 4843.59, "text": "And, basically, if you that's what" }, { "start": 4844.79, "end": 4847.75, "text": "the new TM fix. Aaron, which of them?" }, { "start": 4849.27, "end": 4850.1501, "text": "I'm sorry." }, { "start": 4850.6301, "end": 4852.469999999999, "text": "I'm looking at the chat window. So yeah." }, { "start": 4853.955, "end": 4857.475, "text": "And, again, what you see in saw in July doesn't really" }, { "start": 4861.5552, "end": 4864.5952, "text": "doesn't really make a lot of" }, { "start": 4868.29, "end": 4876.6902, "text": "I don't we had had historically TM with a lot of issues. The latest build are better. If you think if you're talking about the graceful shutdown," }, { "start": 4877.25, "end": 4881.8099999999995, "text": "we had been releasing different updates for it throughout '24" }, { "start": 4881.295, "end": 4884.495, "text": "with the latest one coming in June, I think," }, { "start": 4884.975, "end": 4886.895, "text": "or May, something like that." }, { "start": 4887.215, "end": 4887.9349999999995, "text": "K." }, { "start": 4888.975, "end": 4897.969999999999, "text": "So So just a a follow-up question. So Yep. When I turn the scheduler off, anything that is already" }, { "start": 4898.05, "end": 4899.57, "text": "in inactive" }, { "start": 4899.65, "end": 4902.29, "text": "won't be affected. Right? Because it's already" }, { "start": 4902.29, "end": 4903.09, "text": "triggered." }, { "start": 4903.09, "end": 4913.625, "text": "So it wouldn't okay. Alright. So all you're doing when you turn off the schedule and the folder monitor is you stop the service from going and checking if there is something to run new." }, { "start": 4913.9448, "end": 4921.7047999999995, "text": "Anything that already so if you have 30 files that were supposed to be pulled at 1PM and you shut it down at 01:05" }, { "start": 4921.7047999999995, "end": 4922.665, "text": "or 01:02," }, { "start": 4923.16, "end": 4926.4400000000005, "text": "if they are already in the event queue, they will get executed." }, { "start": 4926.4400000000005, "end": 4931.08, "text": "So if the event was thrown, we'll finish it, but we will not run it again." }, { "start": 4932.360000000001, "end": 4937.8, "text": "So if you shut it down at 01:05 and you have something running every five minutes, the 10105" }, { "start": 4937.8, "end": 4940.0652, "text": "will be skipped because you are down." }, { "start": 4940.7852, "end": 4941.5851999999995, "text": "Okay." }, { "start": 4941.745, "end": 4942.3853, "text": "Just" }, { "start": 4942.625, "end": 4947.0251, "text": "I I don't know. I just find that to be a little bit more manual maybe." }, { "start": 4947.2651000000005, "end": 4950.465, "text": "Because what the Gradescope shutdown is exactly that," }, { "start": 4951.2998, "end": 4968.9448, "text": "except that it won't even pick new jobs from the the event queue while I want it to pick up the additional jobs. So with the file had draft, I want it to finish processing it. Right? While the graceful shutdown is just finishing the current process. See where the difference is. Right? Yes. Yes. So that's" }, { "start": 4968.9448, "end": 4972.785, "text": "one way, especially if your DNS people are doing something anyway." }, { "start": 4973.505, "end": 4974.225, "text": "Mhmm." }, { "start": 4974.625, "end": 4984.0203, "text": "So I don't know. I mean, you can test it and play with that, but most of the updates I always do, I always will do it that way. I'll cut external access early." }, { "start": 4984.9001, "end": 4992.1802, "text": "I'll make sure my folder monitor and my schedule stop picking up new files for a while, you know, especially if I'm in a maintenance window." }, { "start": 4992.1802, "end": 5004.495, "text": "But so one thing to be careful about, JD, is if the maintenance window is because of networking, your outbound will not succeed anyway because they'll probably cut you out. At this point, gradual shutdown is a better idea." }, { "start": 5005.375, "end": 5006.5752, "text": "So you know?" }, { "start": 5006.895, "end": 5008.8150000000005, "text": "Or just shut it down. But the point is," }, { "start": 5009.54, "end": 5013.46, "text": "there is a lot of ways to skin that cat. But if just cleaning" }, { "start": 5013.8598999999995, "end": 5015.780000000001, "text": "the event queue completely," }, { "start": 5016.0999, "end": 5018.74, "text": "you'll lose processing. That's the problem. Yeah." }, { "start": 5019.219999999999, "end": 5028.5251, "text": "I I I understand that. I see. So one thing I like doing, the other thing is look at the tracking table. If nothing is in progress, then look in the event queue." }, { "start": 5028.925, "end": 5030.6849999999995, "text": "If the event queue is empty" }, { "start": 5031.3252, "end": 5035.725, "text": "and the tracking table doesn't have anything in progress, nothing is going on on this server." }, { "start": 5036.780000000001, "end": 5038.0599999999995, "text": "You can shut it down." }, { "start": 5038.3, "end": 5043.98, "text": "But how often does it happen to any of us? Almost ever. Right? There is almost always something in the event queue waiting." }, { "start": 5044.54, "end": 5045.66, "text": "Yeah. Yeah." }, { "start": 5046.139999999999, "end": 5058.3252, "text": "And one of the one of the for for the environment that I'm talking about, we have a lot of folder monitors. We're a a window shop. And so Yep. But if I turn I can see if I turn the folder monitor off" }, { "start": 5059.2852, "end": 5064.485000000001, "text": "Yep. Then it will stop all all new. So this is a good strategy. Thank you." }, { "start": 5065.55, "end": 5073.3901000000005, "text": "And it's just the service that doesn't stop it from actually pulling the files in. It's just stopping the service that is going out and looking for new files." }, { "start": 5073.55, "end": 5074.27, "text": "So" }, { "start": 5075.07, "end": 5083.075, "text": "and once you start it again, it will process whatever else remains. So even if something was just marked but not put in the tracking table" }, { "start": 5083.155000000001, "end": 5087.075, "text": "or in the event queue, it will still pick them up on when you start it again." }, { "start": 5087.395, "end": 5096.04, "text": "So I I don't know. I mean, test it a little bit more, obviously. Yeah. Yes. But I I I generally much prefer that as an option." }, { "start": 5096.52, "end": 5102.2, "text": "Grateful shutdown is great when you do ZGO because you'll never stop. But if you're stopping anyway, good thing in" }, { "start": 5102.52, "end": 5104.2, "text": "inbounds before outbound" }, { "start": 5104.280000000001, "end": 5108.775, "text": "and processing makes a lot more sense for me. I don't know. Okay." }, { "start": 5108.775, "end": 5109.8150000000005, "text": "Thank you. Good." }, { "start": 5110.455, "end": 5111.975, "text": "And we managed to get" }, { "start": 5112.375, "end": 5113.415, "text": "to the time." }, { "start": 5113.655000000001, "end": 5115.575, "text": "Sorry, Nicole. I was trying." }, { "start": 5116.695, "end": 5118.135, "text": "So Don't be sorry." }, { "start": 5119.08, "end": 5121.2402, "text": "Thanks, everyone, for joining us." }, { "start": 5121.8, "end": 5126.9202000000005, "text": "The next one is in four weeks, I think, or something like that." }, { "start": 5128.04, "end": 5129.8801, "text": "And if you" }, { "start": 5129.8801, "end": 5133.655000000001, "text": "have if you are in Chicago, Dallas, or Sacramento," }, { "start": 5133.655000000001, "end": 5139.575, "text": "please come see me in person. We're having in person user groups in all three places in the next couple of months." }, { "start": 5141.4949, "end": 5144.695, "text": "Thanks, everyone, for joining. I hope this was useful." }, { "start": 5145.5599999999995, "end": 5152.68, "text": "If you have any follow-up questions in the meantime, please use community or send us a mail or and so on." }, { "start": 5153.0, "end": 5155.6401000000005, "text": "And Nicole will send you a survey." }, { "start": 5155.88, "end": 5157.08, "text": "I know we're annoying," }, { "start": 5158.275, "end": 5158.915, "text": "but" }, { "start": 5159.2348999999995, "end": 5165.635, "text": "if you want this to keep happening, we need our bosses to know that everyone likes them and what you think about them." }, { "start": 5165.875, "end": 5170.514999999999, "text": "If you have any feedback on the format or anything else, please let us know." }, { "start": 5170.835, "end": 5172.355, "text": "And, Nicole, back to you." }, { "start": 5174.110000000001, "end": 5174.75, "text": "Yes." }, { "start": 5175.07, "end": 5177.55, "text": "Given that it's" }, { "start": 5177.55, "end": 5178.83, "text": "one" }, { "start": 5178.99, "end": 5180.51, "text": "minute after" }, { "start": 5180.75, "end": 5181.87, "text": "the time" }, { "start": 5182.110000000001, "end": 5182.9102, "text": "scheduled," }, { "start": 5182.99, "end": 5188.75, "text": "I won't go through the latest slides that I go through usually about" }, { "start": 5189.205, "end": 5192.6453, "text": "the community portal and what you can get from there" }, { "start": 5193.0452000000005, "end": 5195.8452, "text": "and about the videos on YouTube" }, { "start": 5196.405000000001, "end": 5199.2852, "text": "and also about the g two platform." }, { "start": 5199.2852, "end": 5202.165, "text": "But I will be sharing this presentation," }, { "start": 5202.165, "end": 5205.65, "text": "so please have a look at the presentation" }, { "start": 5206.3699, "end": 5208.45, "text": "that will be posted on the article." }, { "start": 5208.6900000000005, "end": 5211.17, "text": "So thank you all for joining again." }, { "start": 5211.41, "end": 5214.0498, "text": "And the next one is" }, { "start": 5214.0498, "end": 5215.01, "text": "scheduled" }, { "start": 5215.01, "end": 5215.889999999999, "text": "early" }, { "start": 5215.889999999999, "end": 5216.6900000000005, "text": "November." }, { "start": 5217.7449, "end": 5218.545, "text": "I" }, { "start": 5218.545, "end": 5219.665, "text": "believe it's" }, { "start": 5219.665, "end": 5225.425, "text": "November 6, but I'm not so sure if it's six or seven. So I think it's seven." }, { "start": 5226.0650000000005, "end": 5227.585, "text": "Oh, seven. Okay." }, { "start": 5228.2249, "end": 5228.865, "text": "So" }, { "start": 5229.665, "end": 5231.505, "text": "you should have the" }, { "start": 5232.54, "end": 5234.1401000000005, "text": "the way to register." }, { "start": 5234.219999999999, "end": 5239.34, "text": "But if you can't find it, please send me an email, and I will send you the" }, { "start": 5240.1401000000005, "end": 5241.26, "text": "the registration" }, { "start": 5241.26, "end": 5241.9001, "text": "link" }, { "start": 5242.1401000000005, "end": 5245.1802, "text": "that is posted on the community portal, by the way." }, { "start": 5245.985000000001, "end": 5249.1849999999995, "text": "Okay. With that, thank you, Annie. Thank you," }, { "start": 5249.585, "end": 5250.3052, "text": "customers," }, { "start": 5250.7852, "end": 5252.945, "text": "and see you soon. Yep." }, { "start": 5253.344999999999, "end": 5253.985000000001, "text": "Yeah." }, { "start": 5254.225, "end": 5257.2651000000005, "text": "Thanks, everyone, and talk to you next time." }, { "start": 5259.08, "end": 5261.0, "text": "Thanks, Annie. Thanks, Nicole." }, { "start": 5261.5599999999995, "end": 5263.4001, "text": "Bye bye. Have a great day. You" }, { "start": 5263.8801, "end": 5264.76, "text": "too. Great evening." } ] }