[ { "start": 2670.2, "end": 2671.639999, "text": "So prerequisites" }, { "start": 2671.639999, "end": 2672.599999, "text": "for Linux." }, { "start": 2675.964999, "end": 2681.884998, "text": "And over here, things had been updated completely now with the for the new databases." }, { "start": 2684.204998, "end": 2685.645, "text": "Was 30 environments," }, { "start": 2687.759998, "end": 2689.279999, "text": "post grad databases." }, { "start": 2691.759998, "end": 2694.079998, "text": "This is for enterprise clusters," }, { "start": 2694.32, "end": 2697.199997, "text": "and then the admin guide will have something. So" }, { "start": 2698.0, "end": 2702.72, "text": "which what you're looking for which is which ports we're using for the conversation now. Right?" }, { "start": 2704.455, "end": 2706.055, "text": "Yeah. Which ports" }, { "start": 2706.295002, "end": 2706.855, "text": "and" }, { "start": 2708.215, "end": 2709.095, "text": "potentially" }, { "start": 2710.615, "end": 2714.935, "text": "the the the the article that I have that I've been going off of" }, { "start": 2715.735, "end": 2716.775, "text": "has" }, { "start": 2720.27, "end": 2721.79, "text": "it has the ports" }, { "start": 2722.11, "end": 2724.51, "text": "and it says the new packages." }, { "start": 2725.79, "end": 2727.15, "text": "It doesn't say which Oh," }, { "start": 2728.51, "end": 2733.229996, "text": "yeah. These are the two from the release guide. So the release guide have the two packages required." }, { "start": 2733.805, "end": 2747.244995, "text": "If you look, it's it's a couple of lines into the note over there. The there are just two of them. If you are coming from a new enough version of MariaDB, so if you're in the last three months, we already updated before the the the postgre requirement." }, { "start": 2748.59, "end": 2755.78999, "text": "But if you're coming from a lot older, they're in there. But, basically, if you look at the screen over here on the requirement for installing on Linux," }, { "start": 2756.42999, "end": 2759.549995, "text": "those are the ones that we require require" }, { "start": 2759.78999, "end": 2763.149994, "text": "and the two that are really required for Postgre disguise." }, { "start": 2767.865, "end": 2768.985, "text": "The LeapXML" }, { "start": 2768.985, "end": 2770.825, "text": "two and the LeapX SLT." }, { "start": 2771.305, "end": 2777.945, "text": "Gotcha. So what happened when we dropped the September release is that we updated the admin guide and the" }, { "start": 2778.29, "end": 2781.89, "text": "install guide to be for the September release specifically now." }, { "start": 2782.21, "end": 2798.24501, "text": "Although in a couple of places, there are notes if you're on older versions, you need to do that and so on. So see how for The US user route. For example, now we have this line for the PostgresQL requires a new new non route user. This is also in the release notes. So if you're starting brand new," }, { "start": 2799.285, "end": 2803.60501, "text": "you will already know that. But if you're coming from an old one, that's why it's in the update." }, { "start": 2804.005, "end": 2804.645, "text": "Makes sense?" }, { "start": 2805.44, "end": 2806.16, "text": "Okay." }, { "start": 2807.76, "end": 2810.24, "text": "And then on a semi related" }, { "start": 2810.4, "end": 2810.8, "text": "note" }, { "start": 2811.28, "end": 2812.0, "text": "Mhmm. One" }, { "start": 2812.56, "end": 2816.08, "text": "of the ports that's no longer required is the four four four port." }, { "start": 2816.96, "end": 2818.64, "text": "Between the two servers only." }, { "start": 2820.19499, "end": 2821.075, "text": "What was that?" }, { "start": 2821.635, "end": 2827.155, "text": "Oh, between the two servers. Gotcha. You still need it to run because that's where your admin UI runs." }, { "start": 2827.39499, "end": 2836.96, "text": "But back in the day, we're using it for the servlet. Remember at the beginning of the call when I was explaining how we exchange information to the RMI and the servlet, That was the part used for that." }, { "start": 2837.20001, "end": 2838.32, "text": "Okay. Gotcha." }, { "start": 2839.6, "end": 2849.84001, "text": "So that that's why it dropped. But you still need the part itself. And if you go at the very first page on the documentation, there is a very nice checklist of all the parts you need" }, { "start": 2852.225, "end": 2856.145, "text": "that we are I started. So if you go to docsactuary.com" }, { "start": 2856.145, "end": 2860.54501, "text": "prod MFT product, secure transport, there's a checklist with the ports." }, { "start": 2860.705, "end": 2864.56, "text": "It basically lists everything you need. So we don't drop" }, { "start": 2864.56, "end": 2867.84, "text": "any port per se. We actually need to add ports," }, { "start": 2868.0, "end": 2873.2, "text": "but we don't need the four four four or eight four four or whatever you're using for the admin UI" }, { "start": 2873.51999, "end": 2879.76, "text": "to be open between the nodes because the two admins don't talk to each other anymore. They just talk to to to their databases." }, { "start": 2880.375, "end": 2881.655, "text": "That's the big difference." }, { "start": 2882.295, "end": 2885.015, "text": "Okay. Gotcha. Thank you. Yep." }, { "start": 2885.735, "end": 2886.455, "text": "Okay." }, { "start": 2887.015, "end": 2899.08, "text": "The one port that is not in use anymore is the one down on if you are down on the server, there was another port called four four four three or something like that. That's the one we don't need anymore." }, { "start": 2899.64, "end": 2902.6, "text": "The TM communication, the two TMs talking to each other." }, { "start": 2902.92, "end": 2904.28, "text": "The RMI port," }, { "start": 2904.43999, "end": 2905.88, "text": "that's the one that drops," }, { "start": 2907.23999, "end": 2908.43999, "text": "but not the four four four." }, { "start": 2912.705, "end": 2913.66501, "text": "Makes sense?" }, { "start": 2915.58499, "end": 2917.745, "text": "Yes. Thank you. Okay." }, { "start": 2918.865, "end": 2921.185, "text": "I'm seeing a raised hand. Jean?" }, { "start": 2922.54, "end": 2923.74002, "text": "Good morning." }, { "start": 2924.38, "end": 2926.86002, "text": "Along that line, the" }, { "start": 2930.14, "end": 2931.02002, "text": "document" }, { "start": 2931.26, "end": 2935.66, "text": "that you just I've been going through the getting started guide because we're installing," }, { "start": 2936.06, "end": 2937.02002, "text": "net new" }, { "start": 2937.645, "end": 2941.565, "text": "Yep. To our dev servers now. What was the document that" }, { "start": 2941.565, "end": 2945.165, "text": "you just mentioned that lists all the ports? Because I've been trying to go through" }, { "start": 2945.72498, "end": 2946.76498, "text": "and and" }, { "start": 2947.00497, "end": 2948.365, "text": "get all my ports" }, { "start": 2948.76498, "end": 2958.84, "text": "so I can get my firewall guys to open them. Okay. And I've just run into yet another port that I did not have. Okay. So go to It's a little frustrating." }, { "start": 2958.91998, "end": 2959.56, "text": "Yeah." }, { "start": 2959.7200000000003, "end": 2963.0, "text": "Can you see my screen? So from the docs portal," }, { "start": 2963.24, "end": 2964.76, "text": "manage file transfer," }, { "start": 2965.0, "end": 2966.03998, "text": "secure transport." }, { "start": 2968.875, "end": 2970.315, "text": "Getting started." }, { "start": 2972.715, "end": 2973.51498, "text": "And" }, { "start": 2973.75497, "end": 2976.715, "text": "start working with the the initial configuration." }, { "start": 2977.435, "end": 2979.35498, "text": "And over here, there is a checklist." }, { "start": 2979.835, "end": 2980.1549999999997, "text": "Mhmm." }, { "start": 2981.08002, "end": 2983.24002, "text": "And over here are" }, { "start": 2983.56003, "end": 2991.80002, "text": "a gazillion and 17 ports for different things. Now Okay. Do you see seven the one we just ran into now is 7475." }, { "start": 2992.6, "end": 2995.24002, "text": "Oh, this one. Do we know about that one?" }, { "start": 2995.915, "end": 2997.11502, "text": "You" }, { "start": 2997.11502, "end": 2998.23502, "text": "do? I didn't." }, { "start": 2999.27502, "end": 3001.83502, "text": "ACN somewhere else at 7475." }, { "start": 3001.83502, "end": 3003.27502, "text": "Right? Yeah." }, { "start": 3004.23502, "end": 3005.59503, "text": "And it's in the" }, { "start": 3006.635, "end": 3008.635, "text": "it's in the admin guide I just found." }, { "start": 3009.3, "end": 3030.665, "text": "So I found a whole lot of ports, but is there a place that will list everything I need to begin with? Because it's certainly not wasn't fully in the getting started guide. Yeah. We have a workbook, and I don't know if it's it's not public, though. When when PSO are doing installs, we have our all the old workbook where we basically send it over." }, { "start": 3031.305, "end": 3042.7799999999997, "text": "I am not sure what happens when someone tries on their own. Let me take a note, I'll check, and I'll get back to you. So the answer is we know the information. I don't know if it's somewhere on a public place, so I just need to check." }, { "start": 3043.97998, "end": 3044.86, "text": "Oh, okay." }, { "start": 3045.65997, "end": 3052.53998, "text": "Thank you. I'll I'll let I'll let Byron know that too, by the way, so we can maybe get Yeah. So because this is frustrating." }, { "start": 3054.695, "end": 3066.65, "text": "Drop me you know, drop me a note if you don't hear from me in the next couple of days if I get distracted or if I if people don't respond to me and I forget to chase them. It's Yeah. I sure will. It's" }, { "start": 3067.05002, "end": 3074.09003, "text": "I know that we have the workbook because we were just reviewing it, and I'm pretty sure we have the list somewhere in a public place as well." }, { "start": 3074.57, "end": 3080.25, "text": "I just cannot it's probably somewhere on the support side. I just need to dig it out, and they don't want to spend half the call digging out." }, { "start": 3081.23502, "end": 3081.87503, "text": "And" }, { "start": 3082.11502, "end": 3085.39502, "text": "yeah. You know, work case scenario, I'll just send you the workbook anyway." }, { "start": 3086.035, "end": 3087.23502, "text": "Perfect. Thank you." }, { "start": 3087.71503, "end": 3093.315, "text": "Okay. But let me also verify internally and see what we want to do with that. That's the more important part." }, { "start": 3093.96002, "end": 3094.6, "text": "So" }, { "start": 3094.84, "end": 3101.48, "text": "but yes. And there is also a couple of high ports that are not listed there. They're in in the 9,000" }, { "start": 3101.48, "end": 3103.80002, "text": "range, nine six something or another." }, { "start": 3104.04, "end": 3106.12, "text": "You're you're still going to install with Oracle. Right?" }, { "start": 3106.815, "end": 3107.61502, "text": "Correct." }, { "start": 3108.0950000000003, "end": 3113.855, "text": "Yeah. There are a couple over there for the caches that we're using for the caching that are the range for the cache" }, { "start": 3114.575, "end": 3116.73502, "text": "that also get missed occasionally." }, { "start": 3117.295, "end": 3118.415, "text": "So" }, { "start": 3119.88, "end": 3125.56003, "text": "ping me. I'll I'll get you the complete set of ports, and then we'll figure out how to get that published somewhere." }, { "start": 3126.76, "end": 3129.80002, "text": "Perfect. I'll email you now. Mhmm." }, { "start": 3130.36002, "end": 3130.92, "text": "Okay." }, { "start": 3133.195, "end": 3135.515, "text": "Okay. What else do we have?" }, { "start": 3142.315, "end": 3143.115, "text": "No?" }, { "start": 3144.235, "end": 3149.035, "text": "Okay. Had anyone been playing with the new route packages" }, { "start": 3149.65, "end": 3150.77002, "text": "by any chance?" }, { "start": 3152.77, "end": 3153.81, "text": "Hey, Annie." }, { "start": 3154.37, "end": 3155.17, "text": "Hey." }, { "start": 3155.81, "end": 3162.05002, "text": "Oh, yeah. Not related to what you've literally just said, but I was just trying to find the unmute button at the time." }, { "start": 3163.57, "end": 3163.81, "text": "So" }, { "start": 3165.13498, "end": 3169.13498, "text": "we're seeing we're we're on a we're we're on 5.5," }, { "start": 3169.13498, "end": 3170.175, "text": "but we're on" }, { "start": 3170.815, "end": 3172.335, "text": "a much older patch level." }, { "start": 3172.495, "end": 3174.575, "text": "We're we're still in 2022," }, { "start": 3174.575, "end": 3177.855, "text": "so we're we're quite behind the times in that respect." }, { "start": 3178.99, "end": 3180.19, "text": "But we're seeing" }, { "start": 3180.67, "end": 3181.79, "text": "issues with" }, { "start": 3182.11, "end": 3185.1499599999997, "text": "when we hit the file tracking button" }, { "start": 3185.4700000000003, "end": 3187.4700000000003, "text": "to look at the file tracking log," }, { "start": 3188.91, "end": 3190.82996, "text": "it can it's taking sort of" }, { "start": 3191.475, "end": 3194.435, "text": "it can take sort of, like, twenty seconds or so," }, { "start": 3195.63495, "end": 3198.115, "text": "sometimes a bit more to actually come back" }, { "start": 3198.5149499999998, "end": 3202.2749599999997, "text": "with the results, but the server log most instant." }, { "start": 3202.355, "end": 3203.475, "text": "Yes." }, { "start": 3203.875, "end": 3207.07495, "text": "How so couple of questions. What kind of database do you have?" }, { "start": 3207.8999599999997, "end": 3209.5, "text": "It's an external" }, { "start": 3211.66, "end": 3212.62, "text": "SQL" }, { "start": 3212.77997, "end": 3213.58, "text": "database." }, { "start": 3214.1399499999998, "end": 3215.1, "text": "Okay. How" }, { "start": 3215.5, "end": 3218.2200000000003, "text": "many days do you keep in the tracking table?" }, { "start": 3219.18, "end": 3221.82, "text": "So in the file tracking, we got ninety days." }, { "start": 3222.695, "end": 3224.295, "text": "Ninety? Nine zero?" }, { "start": 3224.455, "end": 3225.895, "text": "Yeah. Nine zero." }, { "start": 3226.215, "end": 3228.695, "text": "How many files do you have per day?" }, { "start": 3230.855, "end": 3232.455, "text": "Not that many considering" }, { "start": 3232.855, "end": 3236.135, "text": "what some of these guys on here probably do." }, { "start": 3237.015, "end": 3238.135, "text": "I think" }, { "start": 3238.67, "end": 3239.4700000000003, "text": "we" }, { "start": 3241.87, "end": 3247.07, "text": "well, I do per month, it's about half a million, so it's not actually that many per day." }, { "start": 3247.79, "end": 3248.75, "text": "So Yeah." }, { "start": 3249.4700000000003, "end": 3252.91, "text": "It was. So your problem is that you're keeping too many days." }, { "start": 3253.855, "end": 3256.6549999999997, "text": "So the way our this" }, { "start": 3257.215, "end": 3262.09503, "text": "data bay the the file tracking database is not really designed to keep" }, { "start": 3262.415, "end": 3270.24, "text": "a lot of information in. And because of the way we're doing our indexing and so on based on the days with ninety days in," }, { "start": 3270.56, "end": 3277.2, "text": "it's that's part of your problem. So you have about one and a half million records, which is not that many." }, { "start": 3277.75995, "end": 3282.48, "text": "But even on that volume, I would start with lowering the days a little bit, honestly." }, { "start": 3284.245, "end": 3285.365, "text": "You can also" }, { "start": 3285.685, "end": 3288.725, "text": "see how much memory do you have in the admin UI." }, { "start": 3290.645, "end": 3294.885, "text": "Do you I I yeah. I don't know off the top of my head on that one." }, { "start": 3295.205, "end": 3304.0, "text": "I know it's in the configuration, but off top of my head, I'm not sure. Yeah. Get it up at least up to four gigabytes, if not higher, four to six." }, { "start": 3304.32, "end": 3312.0, "text": "See if that helps a little bit. But with that number of days kept in the database, the tracking table will always be slower than usual." }, { "start": 3312.815, "end": 3317.855, "text": "It's just we're we're not optimized. Our advice is no more than thirty days." }, { "start": 3318.415, "end": 3322.735, "text": "Forty five for very small environments, your yours will qualify in a way." }, { "start": 3323.535, "end": 3331.04004, "text": "I would so it's not just the number of records. So because based on the number of records, I'm okay with you keeping almost 90." }, { "start": 3331.2, "end": 3334.72003, "text": "Right? With the 1 and a half mil, that's not a problem usually," }, { "start": 3335.36, "end": 3337.6000400000003, "text": "but it's just not optimized" }, { "start": 3337.6000400000003, "end": 3342.56, "text": "for going to that many days and months because of how it's done behind the scenes. And" }, { "start": 3343.255, "end": 3353.255, "text": "the other thing is and you'll know I'll say that, update your server. We had been doing a lot of work on this table optimizing it. So the newer version should work a little faster." }, { "start": 3353.575, "end": 3357.415, "text": "But still, I I strongly recommend not to keep ninety days." }, { "start": 3359.19, "end": 3363.75, "text": "Alright. That that's that's really helpful. Thank you, Annie. So one of the reasons" }, { "start": 3364.70996, "end": 3367.26996, "text": "we felt the need to have" }, { "start": 3367.43, "end": 3369.67, "text": "that number of days is because we sometimes" }, { "start": 3370.385, "end": 3374.865, "text": "have our external users coming in and asking us questions, which" }, { "start": 3375.665, "end": 3379.825, "text": "can sometimes mean that we need to look at the data" }, { "start": 3380.145, "end": 3381.985, "text": "older than thirty days old" }, { "start": 3383.0249599999997, "end": 3384.945, "text": "to do some analysis work and stuff." }, { "start": 3386.44, "end": 3391.24005, "text": "Well So one of the things that we are looking at doing is is the possibility of" }, { "start": 3391.56006, "end": 3393.24005, "text": "sending our logs to" }, { "start": 3393.56006, "end": 3394.44006, "text": "Splunk" }, { "start": 3394.52, "end": 3403.2649499999998, "text": "and do some analysis there rather than just keeping it in the database. Would is that, like, a sort of a thing that you'd recommend rather than keeping it all in the database?" }, { "start": 3403.90497, "end": 3413.58496, "text": "So the tracking table is not really locks. It's data objects. So sending the tracking data to Splunk will not be automatic. You can send the server locks to Splunk without trouble." }, { "start": 3414.26, "end": 3418.66003, "text": "But if you want to send the tracking table data, you'll need to first read it somehow." }, { "start": 3419.54004, "end": 3421.46, "text": "One of the ways through the APIs." }, { "start": 3421.94, "end": 3424.5, "text": "Another way is to look at" }, { "start": 3424.74, "end": 3425.54004, "text": "Sentinel," }, { "start": 3425.54004, "end": 3428.26, "text": "our other server, the one that is basically can" }, { "start": 3428.585, "end": 3431.86505, "text": "keep tracking records for as long as you want." }, { "start": 3432.10504, "end": 3433.86505, "text": "It's basically designed for that." }, { "start": 3434.34503, "end": 3440.825, "text": "And we have the new event router that now can send to Kafka, and they are also looking at other options where it can send." }, { "start": 3441.145, "end": 3443.32, "text": "So that maybe downstream," }, { "start": 3443.32, "end": 3444.76, "text": "that will be the next option." }, { "start": 3445.56, "end": 3455.88, "text": "But, yes, short term, if you create something that consumes the tracking table and copies it to Splunk or sends it to Splunk, that might be the cleanest way to do it for now." }, { "start": 3456.855, "end": 3462.615, "text": "Is it is it possible to because I know through the Log four j configuration," }, { "start": 3463.415, "end": 3465.975, "text": "we can write Yeah. The file." }, { "start": 3466.695, "end": 3469.895, "text": "Can we is it possible to write the file tracking to" }, { "start": 3470.13, "end": 3473.73004, "text": "a file and then just one can test that? Nope." }, { "start": 3473.89, "end": 3477.01, "text": "What kind of information do they usually look for it for?" }, { "start": 3478.5299999999997, "end": 3479.33, "text": "It's" }, { "start": 3480.13, "end": 3481.73, "text": "when they're looking for" }, { "start": 3484.46497, "end": 3487.9049999999997, "text": "issues that they may have had submitting files to us." }, { "start": 3488.865, "end": 3493.3450000000003, "text": "And sometimes they're very delayed in asking these questions," }, { "start": 3493.425, "end": 3499.88, "text": "which is frustrating for us, but Yeah. We have to deal with it because they're we kind of, like, supply a service to them. So" }, { "start": 3500.68, "end": 3501.4, "text": "Okay." }, { "start": 3501.56, "end": 3505.4, "text": "Well, as I said, you might want to look to the transfers API" }, { "start": 3505.56, "end": 3509.08, "text": "that will allow you to export the tracking table in a JSON format," }, { "start": 3510.755, "end": 3516.755, "text": "or you can just export the tracking table in an Excel format directly from the UI, especially with your number of records." }, { "start": 3516.91504, "end": 3522.35504, "text": "And then this information can be saved somewhere, or you can pass it and send it somewhere daily, for example." }, { "start": 3523.02, "end": 3526.54004, "text": "So, for example, keep the last thirty days in the database," }, { "start": 3526.94, "end": 3527.74, "text": "and then on" }, { "start": 3528.38, "end": 3531.1000400000003, "text": "for the last five days, just pass the information" }, { "start": 3531.42004, "end": 3537.34, "text": "and send it to Splunk or another database or somewhere out of ST, out of the operational days ST." }, { "start": 3537.945, "end": 3540.10504, "text": "That might be your cleanest way maybe." }, { "start": 3540.665, "end": 3543.2250400000003, "text": "Okay. So log four j is not an option?" }, { "start": 3543.465, "end": 3548.9049999999997, "text": "No. It's not a log four j lock. That's the problem. It's not a log file at all. We call it a transfer log," }, { "start": 3549.145, "end": 3550.34503, "text": "but that's deceiving." }, { "start": 3552.11, "end": 3563.39, "text": "Right. Yes. That's probably that's probably where our confusion is then. Oh, that's lovely. Alright. Thank you, Annie. Yeah. And if so are you on prem or are you in oh, you're on prem with this version?" }, { "start": 3564.065, "end": 3565.3450000000003, "text": "Yeah. We are. Yeah." }, { "start": 3566.065, "end": 3568.785, "text": "Do you have entitlement for Sentinel by any chance?" }, { "start": 3571.58496, "end": 3573.985, "text": "It's it's something that I think we've" }, { "start": 3574.46497, "end": 3576.305, "text": "kind of looked at before." }, { "start": 3576.945, "end": 3581.59, "text": "I I'd I'd love to expand our Axewate products, but I'm" }, { "start": 3581.59, "end": 3582.87006, "text": "sort of limited" }, { "start": 3583.43005, "end": 3587.83, "text": "with budget and stuff like that. So I'm not sure that would be an option." }, { "start": 3588.47003, "end": 3594.79004, "text": "Well, maybe not short term, but I would say look at it again if you want to because it's exactly for this kind of scenarios" }, { "start": 3595.055, "end": 3601.935, "text": "because it can keep it it's essentially a huge database that you can send all the tracking information. You need goals in real time," }, { "start": 3602.65497, "end": 3606.815, "text": "and you can get all the information even years later if need be." }, { "start": 3607.83, "end": 3608.71, "text": "So that" }, { "start": 3609.75, "end": 3625.555, "text": "and I'm don't get me wrong. I I'm not saying that is the only option. I'm just mentioning that that's part of the reason why we will never redesign tracking table to hold that much information is because our policy is for the longer term, GoSentinel or something else," }, { "start": 3625.795, "end": 3633.875, "text": "go out. But we're also working on a couple of connectors to the event router so that we can actually push to other engines, not just ours." }, { "start": 3633.955, "end": 3635.475, "text": "But we are not not there yet." }, { "start": 3637.35, "end": 3641.83, "text": "Hey, Andy. Are are you Yeah. Are people still writing out to the expert log?" }, { "start": 3642.47003, "end": 3648.87, "text": "We are. That's why I asked what kind of information he needs. The problem with expert log is that it's not connected." }, { "start": 3649.0299999999997, "end": 3652.15, "text": "So they you can see the inbound and you can see the outbound," }, { "start": 3652.875, "end": 3655.91504, "text": "but there is nothing to tell you what belongs to what." }, { "start": 3657.195, "end": 3663.91504, "text": "Yeah. So my No. But we we use that in Splunk a lot. And and so if you're just looking to see if you're" }, { "start": 3665.03503, "end": 3666.715, "text": "for folks that call" }, { "start": 3667.115, "end": 3667.835, "text": "us after," }, { "start": 3668.37, "end": 3670.45, "text": "you know, our forty five days," }, { "start": 3671.49, "end": 3674.93, "text": "we have to go look in Splunk for their file names," }, { "start": 3675.41, "end": 3680.29, "text": "and we'll be able to see if it was delivered or not. Yeah. But you're right. I won't get any." }, { "start": 3681.325, "end": 3692.205, "text": "Well, there was a p the PGP decryption error or, you know, something like that. No. But if it is just about did I got the file or not, Jean is right. So, Machu, if you look on the OS level," }, { "start": 3692.525, "end": 3696.685, "text": "there is another file called expert lock. It lives on all servers and all edges." }, { "start": 3698.42, "end": 3705.0599, "text": "Yeah. I I and I found I found that because I thought that might actually be our answer Yeah. A while ago." }, { "start": 3705.54, "end": 3712.065, "text": "But looking in in that log, it didn't seem to show everything that we might actually be looking for." }, { "start": 3712.3050000000003, "end": 3715.9849999999997, "text": "I I get the I get the point of it being there and and" }, { "start": 3716.465, "end": 3719.585, "text": "some of the data that's held in it, but it didn't seem to" }, { "start": 3719.9049999999997, "end": 3723.3450000000003, "text": "have everything that the track the file tracking" }, { "start": 3723.425, "end": 3743.755, "text": "screen would show. No. It doesn't. It doesn't. It it what it does is basically just transfer itself. So it tell you a file called Annie arrived from this IP on this date on this protocol, but it doesn't tell you if Annie originated the transfer or if the file was pulled from Annie. So it's just a pure transfer log." }, { "start": 3743.995, "end": 3753.035, "text": "So but and Jean is bringing it up, and I was getting there as well at some point maybe, or I I discarded it, but Jean is right why I should have brought it up." }, { "start": 3753.8, "end": 3764.0398999999998, "text": "It actually is easy enough to send to Splunk, especially because it's a rotating file. So it rotates overnight, so you can send yesterday's logs cleanly because they're outside of even" }, { "start": 3765.48, "end": 3767.08, "text": "if it doesn't give you everything," }, { "start": 3767.6549999999997, "end": 3769.7349999999997, "text": "at least it gives you the," }, { "start": 3769.9751, "end": 3778.295, "text": "hey. How many files did this user sent on this date? Things like that will actually be there. It will not tell you what happened to the files for the most part," }, { "start": 3778.535, "end": 3782.6951, "text": "but at least you can get the basic curious, did you get my file on Tuesday?" }, { "start": 3783.12, "end": 3784.24, "text": "Things like that." }, { "start": 3784.6400000000003, "end": 3790.48, "text": "Mhmm. Yeah. I I think you're right. I think it's something that we might need to take another look at and" }, { "start": 3791.2799999999997, "end": 3791.9199, "text": "and" }, { "start": 3792.3199999999997, "end": 3795.0398999999998, "text": "and sort of compare that with the kind of" }, { "start": 3795.3599999999997, "end": 3797.6800000000003, "text": "requests and questions that we get from our external" }, { "start": 3798.625, "end": 3804.6249, "text": "if we can answer them from the stuff that's in there. But yeah. No. Thank you, and thank you, June, for your answers." }, { "start": 3805.025, "end": 3812.9449999999997, "text": "And it's not all or nothing. You know? You can always because you you're a service, you can always basically tell him if you're forty days or later," }, { "start": 3813.4601000000002, "end": 3817.2200000000003, "text": "all you can tell you is if we got the file or at least temporary." }, { "start": 3817.38, "end": 3820.66, "text": "And then if you are exporting your server logs" }, { "start": 3821.06, "end": 3822.98, "text": "or if they're on file system," }, { "start": 3823.38, "end": 3838.8049, "text": "once you find the the way I usually work in these cases, if I need to do something like that, I'll hit the extra lock first to see if the file is there and at what time because that gives you a timeline of when to look into the server lock export or server lock files." }, { "start": 3839.125, "end": 3839.4451, "text": "Right?" }, { "start": 3841.27, "end": 3849.7499, "text": "Yeah. Those types names are important. And especially with the newer versions when you get to that civilized versions of ST a k 2024," }, { "start": 3849.99, "end": 3851.19, "text": "I have to say that." }, { "start": 3851.59, "end": 3852.1499, "text": "But" }, { "start": 3852.4700000000003, "end": 3867.3549000000003, "text": "we updated our Xferlock rights. So you can act we actually built a second expert lock as well for the team specifically with a little bit more details, including threat IDs and stuff like that to make it easier to match into" }, { "start": 3867.6749, "end": 3868.475, "text": "server locks." }, { "start": 3869.9099, "end": 3875.9099, "text": "So once you update to a newer server, you can actually weaponize your extra log a lot better." }, { "start": 3876.0699999999997, "end": 3880.0699999999997, "text": "The old one is still there, but we also added more fields for the TM driven one." }, { "start": 3882.635, "end": 3886.5549, "text": "Okay. That's interesting. Yeah. Look out for that on the on the later versions." }, { "start": 3887.0349, "end": 3894.7949, "text": "Yeah. Which will make it easier then to match to server locks that you also probably are sending somewhere or at least you are hiding somewhere." }, { "start": 3895.115, "end": 3895.835, "text": "So it's" }, { "start": 3896.16, "end": 3900.0, "text": "looking for all the information will always be a problem no matter what." }, { "start": 3900.3201, "end": 3900.88, "text": "But" }, { "start": 3901.3599999999997, "end": 3906.96, "text": "and, again, if if all the problem you have is that the database is just a little bit slow," }, { "start": 3907.3599999999997, "end": 3908.0801, "text": "though so" }, { "start": 3908.775, "end": 3912.615, "text": "that's another thing I was going to ask you. When you are looking for this" }, { "start": 3912.8550999999998, "end": 3918.855, "text": "anything, is it always slow, or is it only for if you're looking for a specific date range or specific file name?" }, { "start": 3919.7349999999997, "end": 3921.495, "text": "It seems to be always." }, { "start": 3922.1800000000003, "end": 3927.5398999999998, "text": "And no matter what kind of filtering we put on it Mhmm. Every time we refresh," }, { "start": 3927.62, "end": 3935.7799999999997, "text": "it it it seems to take the same amount of time. It's quite it is, you know, to to ten, twenty, twenty five seconds, something like that to come back." }, { "start": 3936.755, "end": 3953.1599, "text": "So as I said, first of all, get some more memory into it. Another test to do is run it with a query that covers a single day. So run something with give me file give me all the files that are in the last hour or that are something like that, always that is inside of the same day" }, { "start": 3954.0398999999998, "end": 3956.6799, "text": "because we're splitting the logs per day." }, { "start": 3957.0, "end": 3960.1198999999997, "text": "So whenever we are crossing days is when everything will get slow." }, { "start": 3960.44, "end": 3964.6799, "text": "And the generic query that doesn't check that just says, give me everything. You know?" }, { "start": 3965.9249, "end": 3967.1249, "text": "But that's one thing." }, { "start": 3967.4449999999997, "end": 3976.325, "text": "But as I said, ninety days is usually a bit too much. It it you're not over on the records because that's about when I tell people to start thinking team." }, { "start": 3976.645, "end": 3979.2849, "text": "So if all you have is 1,500,000," }, { "start": 3979.4449999999997, "end": 3981.7799, "text": "you should be general yes" }, { "start": 3981.94, "end": 3982.74, "text": "and no." }, { "start": 3983.1400000000003, "end": 3985.8599999999997, "text": "Because the if if you have 1,500,000" }, { "start": 3985.8599999999997, "end": 3993.0599, "text": "records, you probably have about double that lines. So 3,000,000 lines is only it's about about twice as I like to see in this tracking table." }, { "start": 3994.4049999999997, "end": 3994.725, "text": "So" }, { "start": 3995.7650000000003, "end": 3997.9249, "text": "Cool. That's lovely. Thanks, Annie." }, { "start": 3998.245, "end": 3999.0449, "text": "Okay." }, { "start": 3999.285, "end": 4005.0449, "text": "Good luck figuring it out. And, again, update at some point because there were some improvements that will help you," }, { "start": 4005.605, "end": 4008.325, "text": "but also explore outside of the tracking table." }, { "start": 4009.5701, "end": 4016.9300000000003, "text": "Yeah. That that that was gonna be, like, another question if but I I wanted other people to have a chance to answer a question before I ask" }, { "start": 4016.9300000000003, "end": 4020.21, "text": "that one. It was just about the the path to update" }, { "start": 4020.5299999999997, "end": 4023.3301, "text": "from our version and and what we should expect and stuff." }, { "start": 4024.005, "end": 4024.885, "text": "Okay." }, { "start": 4024.885, "end": 4029.605, "text": "So the short version is the officially, we test up to twelve months." }, { "start": 4030.245, "end": 4037.7650000000003, "text": "So you cannot jump more than twelve months on a single update, so you need to do updates, intermediate update to get to where we are." }, { "start": 4039.5699999999997, "end": 4040.45, "text": "However," }, { "start": 4040.45, "end": 4043.6499, "text": "support had been testing some other parts occasionally." }, { "start": 4044.13, "end": 4052.13, "text": "So with as old as you are as your releases, I would strongly recommend just open a ticket to support and ask them for a recommended part." }, { "start": 4054.4849999999997, "end": 4058.9651, "text": "Yeah. That's I think I think that's probably the the best way to go for us at the moment. So" }, { "start": 4059.6050999999998, "end": 4060.885, "text": "Cool. Thanks a lot." }, { "start": 4061.2051, "end": 4071.8599, "text": "Yeah. Because if they some they have r and d behind the scenes, so r and d might have a better idea. They know which builds to skip here and there. Right?" }, { "start": 4072.1, "end": 4075.3799, "text": "But be prepared for multiple update." }, { "start": 4075.62, "end": 4093.815, "text": "The good news is that that updates are fast. The bad news is that you got you said you have enterprise cluster. Right? Yeah. But you have edges. So don't forget that September, you will update the edges to the new database as well. So the September release on the edges the September update on the edges or later will be a little slower." }, { "start": 4095.0950000000003, "end": 4097.495, "text": "So you need to September 20 right?" }, { "start": 4099.02, "end": 4100.5, "text": "That's September" }, { "start": 4100.5, "end": 4101.5801, "text": "24." }, { "start": 4101.5801, "end": 4102.14, "text": "Yes." }, { "start": 4102.62, "end": 4103.4199, "text": "Okay." }, { "start": 4105.42, "end": 4107.1, "text": "Okay. Cool. Thanks, Nay." }, { "start": 4107.5801, "end": 4108.2999, "text": "Absolutely." }, { "start": 4108.78, "end": 4110.22, "text": "Okay. Idi," }, { "start": 4110.62, "end": 4114.78, "text": "TIM graceful shutdown. Yay. My favorite topic. We fixed that." }, { "start": 4116.085, "end": 4121.3649000000005, "text": "So so Edi is asking, can we talk about the transaction manager Graceful shutdown and what happens?" }, { "start": 4121.6849, "end": 4129.2999, "text": "So Graceful shutdown was built five years ago or more as our first attempt at zero downtime," }, { "start": 4129.78, "end": 4131.54, "text": "it was done very stupidly." }, { "start": 4131.54, "end": 4132.98, "text": "It was essentially" }, { "start": 4134.74, "end": 4143.215, "text": "tell TM that it is going to shut down in thirty minutes, so we stop receiving new connections or new jobs. And then exactly thirty minutes later," }, { "start": 4143.9349, "end": 4147.775, "text": "even if something is still running, we'll shut down, but we can shut down earlier" }, { "start": 4147.775, "end": 4154.255, "text": "if something is still going, which was a pain in the behind because if you had a three hours long transfer, it was cut in the middle." }, { "start": 4155.02, "end": 4158.8598999999995, "text": "So when we started working on ZGO this year or last year," }, { "start": 4159.1, "end": 4165.7399000000005, "text": "they realized that, yeah, we need to fix that. So the new graceful shutdown is actually intelligent." }, { "start": 4166.38, "end": 4173.0650000000005, "text": "When you start the Graceful shutdown, the transaction manager stops receiving new jobs from its protocols" }, { "start": 4173.545, "end": 4176.1849, "text": "and stops picking jobs from the database." }, { "start": 4177.385, "end": 4178.1849, "text": "However," }, { "start": 4178.4249, "end": 4182.505, "text": "any job already in the transaction manager is going to be completed" }, { "start": 4182.8201, "end": 4184.34, "text": "before we shut down." }, { "start": 4184.74, "end": 4191.9400000000005, "text": "So if you have a push or a pull going on that might take three more days to finish because you're running a terabyte file from somewhere" }, { "start": 4192.26, "end": 4193.0599, "text": "weird," }, { "start": 4193.86, "end": 4199.5149, "text": "it will still this TM will keep jogging and working only on this specific transfer" }, { "start": 4199.675, "end": 4201.755, "text": "until it finishes completely." }, { "start": 4203.515, "end": 4208.955, "text": "If it fails for some reason, it will not restart. It will put it in the database and it will consider" }, { "start": 4208.955, "end": 4209.755, "text": "it" }, { "start": 4210.14, "end": 4211.02, "text": "finished." }, { "start": 4211.74, "end": 4219.8999, "text": "If it finishes, the processing itself will be thrown to the database, and it will be sent forever. So it's just the current operation that needs to finish." }, { "start": 4220.38, "end": 4226.8150000000005, "text": "If we're in the middle of a advanced routing that has PGP followed by ZIP followed by push or whatever," }, { "start": 4226.975, "end": 4229.7749, "text": "it will do the whole advanced routing part" }, { "start": 4230.415, "end": 4235.455, "text": "and then finish the process so it it can close the advanced routing." }, { "start": 4236.095, "end": 4238.735, "text": "As we know, when you go into deliveries," }, { "start": 4240.07, "end": 4240.63, "text": "if," }, { "start": 4241.4299, "end": 4249.35, "text": "depending on the the scenario, that might mean waiting them out as well, including all of the retries. If the retries are inside the far route," }, { "start": 4249.91, "end": 4261.1649, "text": "obviously, we'll need to wait them out. So this retries will run. But the idea of the graceful shutdown is that it allows the server to finish what it had started without interruptions." }, { "start": 4261.725, "end": 4264.7649, "text": "And this means if a file is coming from an end user" }, { "start": 4265.085, "end": 4268.0449, "text": "and they're uploading or downloading, we're not going to cut it." }, { "start": 4269.24, "end": 4270.4400000000005, "text": "Excellent, CD?" }, { "start": 4271.0, "end": 4271.6399, "text": "Yes." }, { "start": 4272.28, "end": 4274.36, "text": "It's A couple follow ups." }, { "start": 4274.6, "end": 4277.24, "text": "Mhmm. So in the event that," }, { "start": 4277.8, "end": 4280.04, "text": "say, a folder monitor" }, { "start": 4280.04, "end": 4281.88, "text": "is is hung" }, { "start": 4283.1051, "end": 4284.9451, "text": "and you trigger" }, { "start": 4284.9451, "end": 4285.665, "text": "the" }, { "start": 4286.0651, "end": 4287.505, "text": "graceful shutdown," }, { "start": 4287.665, "end": 4289.025, "text": "will that" }, { "start": 4289.425, "end": 4290.225, "text": "hung" }, { "start": 4290.385, "end": 4291.345, "text": "event" }, { "start": 4291.425, "end": 4292.385, "text": "prevent" }, { "start": 4292.7051, "end": 4296.625, "text": "the TM from fully shutting down, or is there, like, a time out" }, { "start": 4299.5, "end": 4302.6199, "text": "time clock on those? It shouldn't," }, { "start": 4302.7, "end": 4305.2599, "text": "but I can guarantee you no one tested that." }, { "start": 4305.659900000001, "end": 4306.3799, "text": "Okay." }, { "start": 4308.14, "end": 4316.335, "text": "I'll be honest here. Right? Because getting it to hang is not that easy, so I'm pretty sure no one tested that. In theory," }, { "start": 4316.895, "end": 4319.7749, "text": "hunk objects should be cleaned up sooner or later." }, { "start": 4320.015, "end": 4336.04, "text": "And because the team is not picking up more jobs, it should be timing out all the threats at this point. And because it is, you know, shut down, it should have passed that folder monitor and schedule work control. If it is the first server, the one that was holding the folder monitor and schedule work services," }, { "start": 4336.36, "end": 4338.68, "text": "they should have during the shutdown," }, { "start": 4339.335, "end": 4344.8549, "text": "the moment when this is issued, the those two services should have flown to another server in the cluster anyway." }, { "start": 4345.1749, "end": 4346.135, "text": "So that should" }, { "start": 4346.775, "end": 4350.695, "text": "that, in theory, should be enough to actually resolve the deadlock." }, { "start": 4352.4501, "end": 4356.21, "text": "Okay. And if that scenario does exist, it actually should" }, { "start": 4356.4501, "end": 4358.13, "text": "show up in the" }, { "start": 4360.2101, "end": 4361.25, "text": "event queue," }, { "start": 4361.41, "end": 4365.3301, "text": "and we should be able to to kill it in" }, { "start": 4366.1951, "end": 4369.955, "text": "in in that portion. Is that am I thinking correctly?" }, { "start": 4370.515, "end": 4374.595, "text": "So if it is in the event queue, it's not being processed at the moment." }, { "start": 4375.0751, "end": 4379.01, "text": "So the event queue has the waiting list. What is waiting?" }, { "start": 4379.3301, "end": 4384.37, "text": "So the moment when you do the graceful shutdown, you're telling you're telling this TM," }, { "start": 4384.4501, "end": 4397.785, "text": "stop picking up from the event queue. So it can still write to the event queue. So if something happens, for example, it gets the file that needs processing, it will put the incoming end into the event queue, but it will not pick it up again because it knows not to." }, { "start": 4398.665, "end": 4402.585, "text": "But if an event is already taken by a database," }, { "start": 4402.9851, "end": 4406.905, "text": "technically, you will not see it in in the event queue. It's in the memories." }, { "start": 4408.91, "end": 4414.909900000001, "text": "Okay. And there are a couple of exceptions, of course. But for the case we are talking about, that's basically" }, { "start": 4415.15, "end": 4421.3899, "text": "it's it's it's the the hanging is not really an event event. It's basically the the" }, { "start": 4421.55, "end": 4423.15, "text": "folder monitor service itself." }, { "start": 4424.325, "end": 4424.965, "text": "And" }, { "start": 4425.365, "end": 4430.5650000000005, "text": "my guess, and, again, haven't been tested, and I don't know how to test it to even confirm." }, { "start": 4430.8049, "end": 4438.245, "text": "But I would expect that the pure process of switching the services into the other node will be enough to resolve the problem." }, { "start": 4438.85, "end": 4443.1699, "text": "Because for all intents and purposes, it looks like this TM just shut down." }, { "start": 4444.6901, "end": 4445.8099, "text": "Oh, okay." }, { "start": 4446.53, "end": 4450.77, "text": "Okay. We had a a we we had" }, { "start": 4452.245, "end": 4454.8049, "text": "an issue that came up when we" }, { "start": 4455.445, "end": 4456.325, "text": "applied" }, { "start": 4456.4048999999995, "end": 4459.6849, "text": "our our the July patch to our production. And" }, { "start": 4461.365, "end": 4463.365, "text": "there were things in the logs," }, { "start": 4464.23, "end": 4467.4299, "text": "errors in the or no. Actually, there were warns in the logs" }, { "start": 4468.15, "end": 4477.51, "text": "that we did not see in our stage environment when we upgrade. Although our stage environment is not not even close to being as busy as our production." }, { "start": 4478.5751, "end": 4479.215, "text": "And" }, { "start": 4479.855, "end": 4485.775, "text": "some of some of them, there were, like, nine events that talked about a heartbeat" }, { "start": 4486.255, "end": 4487.375, "text": "is missing," }, { "start": 4488.3351, "end": 4489.535, "text": "and that kept" }, { "start": 4489.535, "end": 4492.895, "text": "writing in the TM log, like, every ten seconds." }, { "start": 4494.2001, "end": 4495.7999, "text": "And then there was also" }, { "start": 4498.28, "end": 4499.7999, "text": "a a a poll." }, { "start": 4499.9601, "end": 4501.88, "text": "A poll tried to initiate," }, { "start": 4501.9601, "end": 4505.96, "text": "but it said that it was still in process, and we'll wait for the next schedule." }, { "start": 4506.52, "end": 4511.345, "text": "But that never did clear up. So that just kept writing." }, { "start": 4512.0650000000005, "end": 4516.3049, "text": "I think we went in there manually and fixed a couple of them." }, { "start": 4516.705, "end": 4517.5049, "text": "And so" }, { "start": 4518.385, "end": 4519.825, "text": "I was thinking that" }, { "start": 4520.145, "end": 4522.225, "text": "if we use the" }, { "start": 4522.225, "end": 4523.4249, "text": "graceful shutdown," }, { "start": 4523.98, "end": 4526.4598000000005, "text": "those events that were still pending" }, { "start": 4526.86, "end": 4533.74, "text": "or got caught in in that would resolve that. Do you think that that assessment" }, { "start": 4533.74, "end": 4534.6199, "text": "is true?" }, { "start": 4535.5, "end": 4551.005, "text": "Don't know. Okay. I I because just because I don't know the root cause. So the ones that were telling you that the pool is waiting for the next attempt, this one will be the event queue. So for this one, you could have found it and killed it from the event queue." }, { "start": 4551.8099, "end": 4552.45, "text": "Okay." }, { "start": 4552.7699, "end": 4556.6098999999995, "text": "Okay. So that being the case, so if I do a graceful shutdown" }, { "start": 4556.7699, "end": 4558.1299, "text": "on one node," }, { "start": 4559.33, "end": 4565.25, "text": "I should be able to look in the event queue. And if there's anything in there, will I be able to" }, { "start": 4566.215, "end": 4569.095, "text": "delete them having the TM" }, { "start": 4569.415, "end": 4570.2949, "text": "service" }, { "start": 4570.775, "end": 4571.735, "text": "offline?" }, { "start": 4574.135, "end": 4578.0549, "text": "The TM service offline, yes, because you still have the admin service." }, { "start": 4578.135, "end": 4582.6199, "text": "But don't don't forget that the queue is shared between the nodes." }, { "start": 4582.86, "end": 4587.659900000001, "text": "So as long as the other TM is still running, there will be events in that queue," }, { "start": 4588.1399, "end": 4589.9799, "text": "and you don't want to kill those." }, { "start": 4590.7, "end": 4591.4199, "text": "Okay." }, { "start": 4591.659900000001, "end": 4598.945, "text": "Okay. So kind of our oh, go ahead. No. No. Go I was going to say, if you think that you don't have anything pending," }, { "start": 4598.9451, "end": 4602.865, "text": "so you have you don't have any files that are waiting to be pushed and so on," }, { "start": 4603.5851, "end": 4606.145, "text": "if you shut down all the transaction managers," }, { "start": 4606.7201000000005, "end": 4614.48, "text": "the event queue should only contain kings that are supposed to be there. For example, file arrived, but didn't get processed or retries waiting." }, { "start": 4615.2001, "end": 4619.1201, "text": "So if you so anything that you don't expect can be cleaned." }, { "start": 4619.735, "end": 4624.9349, "text": "So you don't need the running TM to clean the event queue because the event queue itself" }, { "start": 4625.735, "end": 4630.455, "text": "is in the database, and the admin UI actually goes directly into the database for it." }, { "start": 4630.775, "end": 4635.71, "text": "The TM is the reader and the writer, but the queue itself is a database table." }, { "start": 4636.35, "end": 4637.07, "text": "Okay." }, { "start": 4637.15, "end": 4640.4299, "text": "So you don't need to change it. So it sounds like" }, { "start": 4640.67, "end": 4642.5099, "text": "when I when I patch," }, { "start": 4643.07, "end": 4645.15, "text": "I should gracefully shut down" }, { "start": 4646.4349999999995, "end": 4647.5549, "text": "the TM" }, { "start": 4647.875, "end": 4653.875, "text": "on one node. Once that's completed, shut down the TM great the graceful shutdown on" }, { "start": 4653.875, "end": 4655.235, "text": "the second node," }, { "start": 4655.715, "end": 4660.275, "text": "and then look in the in the event queue if there's anything in there, delete those things," }, { "start": 4660.95, "end": 4664.7899, "text": "and then go to the server itself and stop" }, { "start": 4665.27, "end": 4665.9899000000005, "text": "all." }, { "start": 4666.3099, "end": 4685.965, "text": "Yeah. Again, Edi, if you do that and the file the if the last operation that happens is a file arriving, the processing will not happen because you'll delete it. So I wouldn't do it that way. I understand where you're coming from, and I know that something got really weird. The idea of the graceful shutdown, it was built specifically for the zero downtime update" }, { "start": 4686.125, "end": 4697.9, "text": "so that we can do the rolling update. Right? In your case, what I would do is just shut down one of the t the TMs together if you're not going to going to do a ZDO," }, { "start": 4698.3, "end": 4710.995, "text": "and just do the regular update. It what you saw in July shouldn't happen. Let's say it like that. And if you delete so if you decide to do it with the Grateful shutdown, don't forget that when the file arrives" }, { "start": 4711.3150000000005, "end": 4712.8351999999995, "text": "with the Grateful shutdown," }, { "start": 4712.915, "end": 4728.4302, "text": "we will throw an event into the database that says this file need processing and will shut down because we finished the arrival part. Right? Mhmm. Mhmm. If you delete it, this file will never get processed. So it will remain in progress in your tracking table forever," }, { "start": 4728.9102, "end": 4734.030000000001, "text": "and it will just sit on the file system. And there is nothing to be done for this file except someone to reupload it." }, { "start": 4735.355, "end": 4747.6748, "text": "That's why I don't want to go and say go delete everything in the tracking, because the event queue is supposed to have events at all times. This is the list of things that still need to happen. If you have a long retry" }, { "start": 4748.71, "end": 4752.3901000000005, "text": "that is on the waiting list, it will also disappear. So things like that." }, { "start": 4752.79, "end": 4753.9102, "text": "Uh-huh. But" }, { "start": 4754.469999999999, "end": 4766.705, "text": "once it starts, if if you have a subscription that is complaining that it's waiting for the next attempt, this is what you need to get the ID and go find it in the tracking in the event queue, and then you can queue it from there manually." }, { "start": 4767.7449, "end": 4768.385, "text": "Okay." }, { "start": 4769.025, "end": 4774.705, "text": "More of a, you know, surgical strike than just take it out everything just because," }, { "start": 4774.945, "end": 4775.745, "text": "otherwise," }, { "start": 4776.145, "end": 4777.105, "text": "things go weird." }, { "start": 4778.280000000001, "end": 4781.0, "text": "Okay. Now if you have the time," }, { "start": 4781.8801, "end": 4787.32, "text": "what I like doing during updates is if you if you're gonna take an outage anyway," }, { "start": 4787.719999999999, "end": 4793.0, "text": "shut down all of the so I would shut down the scheduler and" }, { "start": 4794.195, "end": 4797.1553, "text": "inbound ports about five minutes before I start." }, { "start": 4797.3152, "end": 4799.715, "text": "This will kill all new files coming in," }, { "start": 4801.0752, "end": 4818.21, "text": "and you can also kill the folder monitor at this point so that you leave the TMs to just finish whatever files are in the system at the moment. And then you can monitor the event queue to see what you see is still there before the final stop. So if you're worried about stuck jobs, that's what I would do more than going with the Graysville shutdown for now." }, { "start": 4819.25, "end": 4820.530000000001, "text": "Or that's a great" }, { "start": 4820.8499, "end": 4824.5051, "text": "that is a great strategy. I'm gonna adopt that into my process." }, { "start": 4824.905000000001, "end": 4825.785, "text": "Just kill" }, { "start": 4826.344999999999, "end": 4836.1849999999995, "text": "if you have an hour time, you might as well, while you're preparing, just kill everyone the first ten minutes so that ST can still work these ten minutes, finish whatever it had started," }, { "start": 4836.344999999999, "end": 4838.5051, "text": "but kill all inbound files earlier." }, { "start": 4839.43, "end": 4840.87, "text": "That that's cleaner." }, { "start": 4840.95, "end": 4843.59, "text": "And, basically, if you that's what" }, { "start": 4844.79, "end": 4847.75, "text": "the new TM fix. Aaron, which of them?" }, { "start": 4849.27, "end": 4850.1501, "text": "I'm sorry." }, { "start": 4850.6301, "end": 4852.469999999999, "text": "I'm looking at the chat window. So yeah." }, { "start": 4853.955, "end": 4857.475, "text": "And, again, what you see in saw in July doesn't really" }, { "start": 4861.5552, "end": 4864.5952, "text": "doesn't really make a lot of" }, { "start": 4868.29, "end": 4876.6902, "text": "I don't we had had historically TM with a lot of issues. The latest build are better. If you think if you're talking about the graceful shutdown," }, { "start": 4877.25, "end": 4881.8099999999995, "text": "we had been releasing different updates for it throughout '24" }, { "start": 4881.295, "end": 4884.495, "text": "with the latest one coming in June, I think," }, { "start": 4884.975, "end": 4886.895, "text": "or May, something like that." }, { "start": 4887.215, "end": 4887.9349999999995, "text": "K." }, { "start": 4888.975, "end": 4897.969999999999, "text": "So So just a a follow-up question. So Yep. When I turn the scheduler off, anything that is already" }, { "start": 4898.05, "end": 4899.57, "text": "in inactive" }, { "start": 4899.65, "end": 4902.29, "text": "won't be affected. Right? Because it's already" }, { "start": 4902.29, "end": 4903.09, "text": "triggered." }, { "start": 4903.09, "end": 4913.625, "text": "So it wouldn't okay. Alright. So all you're doing when you turn off the schedule and the folder monitor is you stop the service from going and checking if there is something to run new." }, { "start": 4913.9448, "end": 4921.7047999999995, "text": "Anything that already so if you have 30 files that were supposed to be pulled at 1PM and you shut it down at 01:05" }, { "start": 4921.7047999999995, "end": 4922.665, "text": "or 01:02," }, { "start": 4923.16, "end": 4926.4400000000005, "text": "if they are already in the event queue, they will get executed." }, { "start": 4926.4400000000005, "end": 4931.08, "text": "So if the event was thrown, we'll finish it, but we will not run it again." }, { "start": 4932.360000000001, "end": 4937.8, "text": "So if you shut it down at 01:05 and you have something running every five minutes, the 10105" }, { "start": 4937.8, "end": 4940.0652, "text": "will be skipped because you are down." }, { "start": 4940.7852, "end": 4941.5851999999995, "text": "Okay." }, { "start": 4941.745, "end": 4942.3853, "text": "Just" }, { "start": 4942.625, "end": 4947.0251, "text": "I I don't know. I just find that to be a little bit more manual maybe." }, { "start": 4947.2651000000005, "end": 4950.465, "text": "Because what the Gradescope shutdown is exactly that," }, { "start": 4951.2998, "end": 4968.9448, "text": "except that it won't even pick new jobs from the the event queue while I want it to pick up the additional jobs. So with the file had draft, I want it to finish processing it. Right? While the graceful shutdown is just finishing the current process. See where the difference is. Right? Yes. Yes. So that's" }, { "start": 4968.9448, "end": 4972.785, "text": "one way, especially if your DNS people are doing something anyway." }, { "start": 4973.505, "end": 4974.225, "text": "Mhmm." }, { "start": 4974.625, "end": 4984.0203, "text": "So I don't know. I mean, you can test it and play with that, but most of the updates I always do, I always will do it that way. I'll cut external access early." }, { "start": 4984.9001, "end": 4992.1802, "text": "I'll make sure my folder monitor and my schedule stop picking up new files for a while, you know, especially if I'm in a maintenance window." }, { "start": 4992.1802, "end": 5004.495, "text": "But so one thing to be careful about, JD, is if the maintenance window is because of networking, your outbound will not succeed anyway because they'll probably cut you out. At this point, gradual shutdown is a better idea." }, { "start": 5005.375, "end": 5006.5752, "text": "So you know?" }, { "start": 5006.895, "end": 5008.8150000000005, "text": "Or just shut it down. But the point is," }, { "start": 5009.54, "end": 5013.46, "text": "there is a lot of ways to skin that cat. But if just cleaning" }, { "start": 5013.8598999999995, "end": 5015.780000000001, "text": "the event queue completely," }, { "start": 5016.0999, "end": 5018.74, "text": "you'll lose processing. That's the problem. Yeah." }, { "start": 5019.219999999999, "end": 5028.5251, "text": "I I I understand that. I see. So one thing I like doing, the other thing is look at the tracking table. If nothing is in progress, then look in the event queue." }, { "start": 5028.925, "end": 5030.6849999999995, "text": "If the event queue is empty" }, { "start": 5031.3252, "end": 5035.725, "text": "and the tracking table doesn't have anything in progress, nothing is going on on this server." }, { "start": 5036.780000000001, "end": 5038.0599999999995, "text": "You can shut it down." }, { "start": 5038.3, "end": 5043.98, "text": "But how often does it happen to any of us? Almost ever. Right? There is almost always something in the event queue waiting." }, { "start": 5044.54, "end": 5045.66, "text": "Yeah. Yeah." }, { "start": 5046.139999999999, "end": 5058.3252, "text": "And one of the one of the for for the environment that I'm talking about, we have a lot of folder monitors. We're a a window shop. And so Yep. But if I turn I can see if I turn the folder monitor off" }, { "start": 5059.2852, "end": 5064.485000000001, "text": "Yep. Then it will stop all all new. So this is a good strategy. Thank you." }, { "start": 5065.55, "end": 5073.3901000000005, "text": "And it's just the service that doesn't stop it from actually pulling the files in. It's just stopping the service that is going out and looking for new files." }, { "start": 5073.55, "end": 5074.27, "text": "So" }, { "start": 5075.07, "end": 5083.075, "text": "and once you start it again, it will process whatever else remains. So even if something was just marked but not put in the tracking table" }, { "start": 5083.155000000001, "end": 5087.075, "text": "or in the event queue, it will still pick them up on when you start it again." }, { "start": 5087.395, "end": 5096.04, "text": "So I I don't know. I mean, test it a little bit more, obviously. Yeah. Yes. But I I I generally much prefer that as an option." }, { "start": 5096.52, "end": 5102.2, "text": "Grateful shutdown is great when you do ZGO because you'll never stop. But if you're stopping anyway, good thing in" }, { "start": 5102.52, "end": 5104.2, "text": "inbounds before outbound" }, { "start": 5104.280000000001, "end": 5108.775, "text": "and processing makes a lot more sense for me. I don't know. Okay." }, { "start": 5108.775, "end": 5109.8150000000005, "text": "Thank you. Good." }, { "start": 5110.455, "end": 5111.975, "text": "And we managed to get" }, { "start": 5112.375, "end": 5113.415, "text": "to the time." }, { "start": 5113.655000000001, "end": 5115.575, "text": "Sorry, Nicole. I was trying." }, { "start": 5116.695, "end": 5118.135, "text": "So Don't be sorry." }, { "start": 5119.08, "end": 5121.2402, "text": "Thanks, everyone, for joining us." }, { "start": 5121.8, "end": 5126.9202000000005, "text": "The next one is in four weeks, I think, or something like that." }, { "start": 5128.04, "end": 5129.8801, "text": "And if you" }, { "start": 5129.8801, "end": 5133.655000000001, "text": "have if you are in Chicago, Dallas, or Sacramento," }, { "start": 5133.655000000001, "end": 5139.575, "text": "please come see me in person. We're having in person user groups in all three places in the next couple of months." }, { "start": 5141.4949, "end": 5144.695, "text": "Thanks, everyone, for joining. I hope this was useful." }, { "start": 5145.5599999999995, "end": 5152.68, "text": "If you have any follow-up questions in the meantime, please use community or send us a mail or and so on." }, { "start": 5153.0, "end": 5155.6401000000005, "text": "And Nicole will send you a survey." }, { "start": 5155.88, "end": 5157.08, "text": "I know we're annoying," }, { "start": 5158.275, "end": 5158.915, "text": "but" }, { "start": 5159.2348999999995, "end": 5165.635, "text": "if you want this to keep happening, we need our bosses to know that everyone likes them and what you think about them." }, { "start": 5165.875, "end": 5170.514999999999, "text": "If you have any feedback on the format or anything else, please let us know." }, { "start": 5170.835, "end": 5172.355, "text": "And, Nicole, back to you." }, { "start": 5174.110000000001, "end": 5174.75, "text": "Yes." }, { "start": 5175.07, "end": 5177.55, "text": "Given that it's" }, { "start": 5177.55, "end": 5178.83, "text": "one" }, { "start": 5178.99, "end": 5180.51, "text": "minute after" }, { "start": 5180.75, "end": 5181.87, "text": "the time" }, { "start": 5182.110000000001, "end": 5182.9102, "text": "scheduled," }, { "start": 5182.99, "end": 5188.75, "text": "I won't go through the latest slides that I go through usually about" }, { "start": 5189.205, "end": 5192.6453, "text": "the community portal and what you can get from there" }, { "start": 5193.0452000000005, "end": 5195.8452, "text": "and about the videos on YouTube" }, { "start": 5196.405000000001, "end": 5199.2852, "text": "and also about the g two platform." }, { "start": 5199.2852, "end": 5202.165, "text": "But I will be sharing this presentation," }, { "start": 5202.165, "end": 5205.65, "text": "so please have a look at the presentation" }, { "start": 5206.3699, "end": 5208.45, "text": "that will be posted on the article." }, { "start": 5208.6900000000005, "end": 5211.17, "text": "So thank you all for joining again." }, { "start": 5211.41, "end": 5214.0498, "text": "And the next one is" }, { "start": 5214.0498, "end": 5215.01, "text": "scheduled" }, { "start": 5215.01, "end": 5215.889999999999, "text": "early" }, { "start": 5215.889999999999, "end": 5216.6900000000005, "text": "November." }, { "start": 5217.7449, "end": 5218.545, "text": "I" }, { "start": 5218.545, "end": 5219.665, "text": "believe it's" }, { "start": 5219.665, "end": 5225.425, "text": "November 6, but I'm not so sure if it's six or seven. So I think it's seven." }, { "start": 5226.0650000000005, "end": 5227.585, "text": "Oh, seven. Okay." }, { "start": 5228.2249, "end": 5228.865, "text": "So" }, { "start": 5229.665, "end": 5231.505, "text": "you should have the" }, { "start": 5232.54, "end": 5234.1401000000005, "text": "the way to register." }, { "start": 5234.219999999999, "end": 5239.34, "text": "But if you can't find it, please send me an email, and I will send you the" }, { "start": 5240.1401000000005, "end": 5241.26, "text": "the registration" }, { "start": 5241.26, "end": 5241.9001, "text": "link" }, { "start": 5242.1401000000005, "end": 5245.1802, "text": "that is posted on the community portal, by the way." }, { "start": 5245.985000000001, "end": 5249.1849999999995, "text": "Okay. With that, thank you, Annie. Thank you," }, { "start": 5249.585, "end": 5250.3052, "text": "customers," }, { "start": 5250.7852, "end": 5252.945, "text": "and see you soon. Yep." }, { "start": 5253.344999999999, "end": 5253.985000000001, "text": "Yeah." }, { "start": 5254.225, "end": 5257.2651000000005, "text": "Thanks, everyone, and talk to you next time." }, { "start": 5259.08, "end": 5261.0, "text": "Thanks, Annie. Thanks, Nicole." }, { "start": 5261.5599999999995, "end": 5263.4001, "text": "Bye bye. Have a great day. You" }, { "start": 5263.8801, "end": 5264.76, "text": "too. Great evening." } ]